There has never been a better time to become proficient in Machine Learning, a field that is rapidly transforming the world as we speak. While there are hundreds of tutorials and courses available online, nothing compares to learning directly from the team that pioneered Machine Learning as a Service (MLaaS) in 2011 and continues to implement real-world Machine Learning solutions every day.
How can you get this hands-on experience, you may ask? One of the best ways is becoming a BigML Intern to get a headstart! We are excited to launch our Internship Program, which welcomes advanced students and young professionals to join the BigML Team. BigML is seeking interns during the summer of 2018 and throughout the whole year for the following roles:
- Product Development
- Software Engineering
- IT
- Marketing
- Business Development
To give a taste of BigML Internship projects, here are two examples from past interns:
- Machine Learning Throwdown: a series of 6 blog posts comparing BigML to other platforms
- How-To Create a WhizzML Script: a series of 4 blog posts on writing WhizzML scripts
Every company, big and small, has data and the need to analyze it to transform it into actionable insights. For this reason, Machine Learning is becoming a common and expected tool in business settings. Regardless of your academic background and career aspirations, having a grasp of applied Machine Learning will serve you well in the future. Machine Learning skills are replacing the “proficient in Excel” requirement on resumes, and this progression will only continue in 2018. In fact, we are already seeing job applications that list BigML as a required skill (see minute 1:45).
The BigML platform continues to expand and improve, thanks to our top-notch team that has been international since day one. Our summer 2017 interns were no exception, representing a variety of backgrounds and coming from 4 different countries of origin (China, India, Spain, and the USA). The BigML Team today consists of 35 members from 6 countries, representing 14 different cities. During 2018, BigML looks forward to bringing several interns on board who are eager to learn and put Machine Learning to good use.
If you are nodding your head at this point and thinking, “sign me up!” please see the dedicated Internships page for more details about the program and how to apply. Feel free to contact internships@bigml.com with any questions. Don’t hesitate and start your application today!
On March 23-25, 2018, more than 1,000 Machine Learning practitioners will gather at Machine Learning Prague 2018 to hear from 45 speakers currently working in the Machine Learning field. The event will take place over 3 days with fun and interesting talks, such as “Really Automating Machine Learning”, presented by Dr. Charles Parker, BigML’s VP of Machine Learning Algorithms, who will present what’s behind the scenes of Machine Learning automation, focusing on how to know when your automation is succeeding, and what to do to make it easier. Interested in attending Dr. Charles Parker’s talk or other sessions at Machine Learning Prague 2018? Purchase your ticket today using the code below to get a 10% discount!
We interviewed Dr. Charles Parker to share a glimpse of what’s to come in his talk at ML Prague 2018. Find the excerpts below.
During the last years of your career, you have been focusing on ML automation. What’s the motivation behind this topic?
The main reason is necessity: the amount of training data available to people is increasing. As it does, the model complexity we can consider grows more and more, which means we need new ways of addressing parameterization of all of these very complex models. Nobody, experts included, has the time to figure out, for example, the effects of all of the various parameters of all of the various ways of doing gradient descent. So these automation methods are sort of bound to become de rigueur in the field.
On a personal level, there’s something compelling to me about working on a layer atop all of these parameters. The problem you’re trying to solve when you’re parameterizing an ML algorithm is “what are the best parameters for this algorithm for this data?.” At the automation level, your task is “what is the way to find the best parameters for any algorithm on any data, given finite compute power and time?” To me, there are a whole bunch of interesting questions in there about the distribution of datasets in the real world as well as the distribution of loss functions. What kinds of data do “most people” have, and how to “most people” know when they’ve got a good model? Knowing those priors would make automation a lot easier. However, not only do we not know what they are, but I don’t think we even have a very good idea about how to figure out what they might be. That’s an exciting place to work.
What would you tell those who refuse to join the Machine Learning revolution?
I’m going to surprise everyone and say those people are absolutely right to be suspicious! Machine Learning is harder than you think. Using ML algorithms is fundamentally a different way of interacting with computers than most people are used to, and it takes some time and talent to develop a facility for that interaction. When we train ML algorithms, we’re really programming with data. As such, even though you don’t need to write code, you need to develop a lot of the same habits of mind that a programmer has: Good programmers learn early on to see their code as the machine sees it; good ML practitioners see their data as the machine sees it.
In addition to all of that, it’s also harder to measure the performance of a Machine Learning algorithm than you think, so even when people have done everything right with their data, they sometimes still don’t get what they think they will.
This isn’t an excuse to let it pass you by, though. Forty years ago, computers themselves were difficult to use and strictly the domain of geek enthusiasts. Those who developed the ability to interact with them early on put themselves in a position to succeed in a whole bunch of different careers. I think it’s very possible that Machine Learning will have the same sort of transformative power for those who take the time to learn how to do it right.
You have been developing Machine Learning applications for quite some time. Based on your experience, what would you recommend to the new ML practitioners that are starting now?
The best way to learn is by practicing. I recommend to get some data and start playing around, trying to predict things, introspecting models. If that’s too vague, check out the BigML education videos and try to find data from other sources where you can do the same things we’re doing in the videos. In this way, you get a sense of how Machine Learning data is “supposed to look” which is maybe the first and most important skill when you start with Machine Learning.
There’s this misperception that you have to know how to write code to get started and that just isn’t true anymore. BigML is a fantastic playground for messing with these ideas because you don’t have to write any code or install any software. On top of that, we’ve got an interface that lets you interact with the models you train, which is hard to come by elsewhere. Sure, it sounds like a pitch for BigML, but if I didn’t think it was the best way for non-experts to do Machine Learning, I probably wouldn’t work here!
About the Lecturer
Dr. Charles Parker is the Vice President of Machine Learning Algorithms at BigML. Dr. Parker holds a Ph.D. in computer science from Oregon State University, and was previously a research associate at the Eastman Kodak Company where he applied Machine Learning to image, audio, video, and document analysis. He also worked as a research analyst for Allston Holdings, a proprietary stock trading company, developing statistically-based trading strategies for U.S. and European futures markets.
His current work for BigML is in the areas of Bayesian Parameter Optimization and Deep Learning. One of his recent projects developed at BigML was bringing Deepnets to the BigML platform, the world’s first deep learning algorithm capable of Automatic Network Detection and Structure Search, which automatically optimize model performance and minimize the need for expensive specialists to drive business value. Dr. Charles Parker ran a benchmark test with over 50 datasets representing a variety of use cases and industries. Based on extensive evaluations that considered 10 different performance metrics, each based on 10-fold cross validation, BigML Deepnets came out on top of 30 competing algorithms.
Additionally, Dr. Charles Parker regularly contributes to the BigML blog. Here are a few highlighted for your leisure:
- Get Smarter about the AI Apocalypse
- Machine Learning Automation Beware of the Hype
- How to Put Machine Learning in your Machine Learning
- Advice to a Young Data Scientist – Part 1 and Part 2
- The People Who Would Teach Machines to Learn
- Machine Learning from Streaming Data: Two Problems, Two Solutions, Two Concerns, and Two Lessons
- Everything you Wanted to Know about Machine Learning but were too Afraid to Ask: Part 1 and Part 2
On Tuesday night, the BigML Team presented at the February Willamette Innovators Network (WiN) PubTalk. This event was hosted by the WiN and the Corvallis Benton County Economic Development Office and held at Eat and Drink 101, a cozy restaurant in downtown Corvallis, Oregon.
The night kicked off with networking time, followed by a round of introductions where anyone in the audience could give a 30-second pitch about their company. Next came the main presentation about BigML, which was given by Adam Ashenfelter, BigML’s Chief Data Engineer and Co-founder. Since we’re used to presenting to ML newbies, Adam first explained what Machine Learning is in a nutshell, and how it’s used in many industries for tasks like fraud detection, market segmentation, recommendation systems, and more. He touched on “the Good and the Bad of Machine Learning” and how BigML is working to solve these challenges:
- The GOOD: thanks to cheap computation, data is abundant for all sorts of industries.
- The BAD: many Machine Learning tools are difficult to use and require deep expertise. To boot, there are not enough ML experts to go around.
- BigML’s SOLUTION: make ML tools 10x easier so everybody can get more from their data and make better decisions.
Adam also shared about BigML’s background and our “locally grown Machine Learning.” As mentioned in our previous post about why BigML is in Corvallis, the BigML Team has strong roots in Corvallis, with seven team members currently living there (three of which are long-time locals). Oregon State University has played a big role in the founding of BigML, thanks to its leading programs in Artificial Intelligence and Machine Learning.
Attendees represented a variety of industries and profiles within the local community including engineers from software startups, professors and students from Oregon State University, members from non-profit organizations, entrepreneurs, and other business people from Corvallis and other cities in Oregon such as Portland and Bend. Twitter was the largest company represented in the audience with a remote worker from Portland.
The crowd was engaged, asking questions about how Machine Learning could be applied to their current businesses. Tom Nelson, Economic Development Manager at the Corvallis Benton County Economic Development Office, commented:
“WiN had an impressive turnout at the PubTalk. We had standing-room only with over 60 in attendance, and we had a VERY cool company, BigML, to tell us about their company and Machine Learning. The topic was obviously a draw!”
Our team in Corvallis extends a big thank you to the Willamette Innovators Network for organizing this event, and we look forward to building more connections with the local community. If you are in the Corvallis area and would like to learn more about BigML, please reach out to us at info@bigml.com. Check out our photo album on Facebook and on Google+ to see more pictures from the event.
There’s something nice about all this media attention being given to the AI these days. It’s nice for one’s work to be seen as important or world-transforming. It’s also nice to see people speaking up to say that it might be risky or dangerous. The dialog about the potential benefits and pitfalls of AI seems to be honest and genuine.
Unfortunately, “honest” and “genuine” doesn’t remotely equate to “nuanced” or “deep”. This is especially true around the so-called “AI apocalypse”, the idea that a general, super-human computer intelligence is inevitable and what that means for the human race. There are at least three opinions on the matter:
- Super-human AI is coming, and it’s going to destroy us all
- Super-human AI is coming, and it’s going to be awesome
- Super-human AI isn’t coming, and you’re naive for thinking it will
The first of these gets the most press, (because who doesn’t love a good apocalypse?) and is espoused by guys like Stephen Hawking and Elon Musk. The second is the singularity crowd, led by Ray Kurzweil and others. The third is probably where most researchers and practitioners sit, including roboticist Rodney Brooks, who did a decent interview about the subject.
The thing that makes the discussion in the popular media depressingly shallow is that rarely is robust reasoning given for why someone thinks their idea is correct. With little in the way of a real argument presented for or against any theory, most readers will doubtlessly rely on their own intuitions, preferences, and fears to guide them. This does not an informed populace make.
Luckily, Dr. Stuart Russell is here to save us from our own ignorance. At a conference I attended recently, he gave a great talk entitled “Provably Beneficial AI” which gave concrete technical arguments that speak to each of the three views above: Why general AI might (or might not) be on its way, and why AI might (or might not) be a significant risk to the human race. While his talk didn’t give too much in the way of answers, it was clear that at least that he’s done a lot of thinking about the questions.
Here’s my summary of that talk. Let’s think along with him, shall we?
The Specter of Artificial Intelligence
The argument from the “Strong AI is inevitable” crowd goes something like this: Look at how much progress AI has made in the last 30 (or 20 or 10 or 5) years! I bet in the next n years it will be MOAR BETTER.
This argument is unconvincing to many in the field, and for good reasons. One is that the things we’ve gotten good at in the last few decades are really just more impressive versions of things we could already do. We’ve been doing limited domain speech recognition fairly well for at least 20 years. Now we’ve got more compute power, more data, models that are a bit more clever, et voila, our domain has expanded to the whole language. Similarly, a pro-level backgammon player was developed 25 years ago; some more compute resources, a bit more cleverness, and we can do the same with chess, and even more recently with Go.
Outside observers of this situation tend to overgeneralize. The idea seems that we have technology to solve backgammon and we “fancied it up” and were able to do well at Go, which is harder. Well, if we “fancy it up even more” we should be able to solve anything, right?
No. There are fundamental differences between speech recognition and game playing, and, well, life in general. Here are a few things that computers can’t yet do that you absolutely need for general intelligence:
Long-term, multi-level planning. Even simple goals we have as people require multiple plans on multiple levels all being attended to simultaneously. A good example is my recent travel from Chicago to Melbourne. This involved two flights, two car rides, two train rides, and loads of walking. Only a few of those individual steps were actually in the direction of my destination because there are many intermediate goals to be satisfied at every step (leave the house, get into the car, leave the driveway, get to the freeway, and so on). Humans are surprisingly good in formulating these ad-hoc, interacting hierarchies of goals in a way that computers are very much unable to do, as of yet.
Generalization of knowledge. Autonomous driving is basically a solved problem, but if you put a system trained for one vehicle into another, the likelihood of failure is pretty high even if the systems have only trivial differences. There aren’t yet any compelling general algorithms that allow computers to automatically reuse parts of already-learned tasks to solve a novel problem. There are parts of Go and Chess that are highly related (the ideas of attacking and defending, for example), but it’s hard to imagine how a chess-trained computer could use that fact to learn how to play Go more quickly. This ability to “analogize” between domains is one of the central pieces of human intelligence.
Real natural language understanding. Right now, you can ask Siri “What’s the weather going to be like today?” But suppose you asked, “Can I go golfing at 5pm today?” There’s a lot more to unpack in even that simple statement. Is the course open? Is there a public course nearby? Am I a member at a nearby private course? What will the weather be like? What is my schedule like? Do I own clubs? Current systems can only answer questions when the answers are fairly straightforward; doing anything more will require a significant leap in complexity over current language models.
Importantly, none of these things will be solved by more computers or bigger computers or quantum computers. These are things that need to be solved on a conceptual level; more computing power will just get us the same wrong answers we have now, faster. There is nothing substantial right now in the research to indicate that these will be solved in a general way in the near future.
Moreover, for some of these problems, there’s no obvious need for a computational solution. After all, how deep of an understanding of natural language does a computer need to have to do everything we want it to do? To have a computer read and really understand a novel, for example, is probably not terribly useful. There are lots of technologies out there that are more than possible from an engineering standpoint (fuel cell vehicles, travel to the moon, mile-high buildings), that never become commonplace only because of economics and their level of necessity.
So anyone who says strong AI is “inevitable” runs up against obvious counter-arguments. We can’t do it now, and there’s no reason now to believe we ever will . . . well, maybe not no reason. Russell shares the cautionary tale of Ernst Rutherford, who in 1933 was quoted in a newspaper article about the impossibility of extracting energy from an atom. Leo Szlisard read the article and, within a day, had conceived of a neutron-initiated nuclear chain reaction. The problem had gone from “impossible” to “essentially solved” in less than 24 hours. Is there any guarantee that strong AI will ever happen? Absolutely not. But it’s a fool’s bet to be on the wrong side of human ingenuity.
Will It Be Awesome or the End of Humanity?
So let’s suppose, for arguments sake, that we do invent a strong form of AI. Why should we believe the alarmists that it will be “evil” and destroy us all?
Certainly, there’s the possibility that an evil person gets a hold of such an AI and uses it against his or her enemies. But that threat isn’t a problem particular to AI; even if AI is an existential threat in such a situation, it’s difficult to make the case that it’s more dangerous than nuclear weapons, which basically have the human race at a hair’s breadth from annihilation all the time. There will always be dangerous things and evil people in the world, with or without AI, and that reality will always have to be dealt with.
But there are problems that are unique to AI, and many of them relate to the fact that a superhuman intelligence would be very good at doing what it was told.
Consider: You tell your super-intelligent robot to go and get you some coffee. What if the only coffee within 10 miles costs $50? You didn’t say to the robot, “Don’t pay too much”, so of course it happily hands over the $50 because that’s what you wanted. What if there’s no coffee shop within 100 miles? Coffee road trip!
Probably you can see how this quickly gets sinister. What if the coffee shop is closed? The robot is happy to break in and make the coffee itself. You wanted coffee, didn’t you? What if a police officer sees that the shop was broken into and tries to pull the robot out of the shop. Well, you said you wanted coffee. Maybe most importantly, what if anyone tries to power down or otherwise destroy the robot? Oh no, you don’t! I can’t get the coffee if I’m dead!
We don’t need to program our robot specifically to engage in violent behavior or over-consumption of resources. If a robot has a goal, and such behavior furthers that goal, the robot might engage in such behavior because it has no reason to avoid it.
The problem is one that Russell calls “value misalignment”. By telling the robot to get the coffee, we’ve implicitly given it a system of values: “Getting the coffee” has some positive value in the robot’s mind. Everything else – time, money, human life, and anything else we ourselves might value – is tied at zero.
Furthermore, the astute reader will see that the problem only gets worse as the robot gets smarter. Some of the most important attributes of human intelligence are adaptation, improvisation, and creativity. We want a robot to be able to figure out how to accomplish it’s goals even if there are obstacles in the way. As it gets more clever, though, its space of possible actions becomes larger and larger, and the more likely it is that its actions will, just by chance, run counter to a human value that it doesn’t know about.
To a degree, we see this in Machine Learning applications already: We’ve become so good at solving optimization problems that if there is a way to “cheat” by being myopic about the problem the optimizer will find it. A good example is the recent debacle where a Google image recognizer labeled some people as gorillas. To the system, the difference between a human face and a gorilla face is fairly subtle, and as long as it gets that right most of the time it thinks it’s doing a good job. After all, if I get all of the gorilla faces right and pull in just a few humans, that’s still a pretty good result, right? But of course, that’s not right at all.
If we told the system that it was important not to make this mistake, it could avoid making it. But if we haven’t told it so, why on earth would it try to avoid this any more than, say, mixing up a microwave and a toaster oven? This can all be summarized succinctly by one of my early CS professors:
“Don’t you hate it when when the program does what you told it to do instead of what you want it to do?”
Clarity of intention is of the utmost importance in computer programming, and shame on us when it is found lacking.
Robot Morality
So, then, what does “telling the robot to avoid this sort of mistake” look like for general behavior in the world? Well, for starters, it means simply encoding the knowledge that human life, compliance with the law, and finite resources like money and time all have a positive value. This by itself goes a long way towards giving the robot a “more human” way of reasoning about its methods.
However, anyone who’s read Asimov knows the danger of relying on rules to govern robotic behavior. Again, computers are great optimizers; if its goals are furthered by finding loopholes in those rules, the robot will find them. It’s the aggressiveness of the optimization itself that needs to somehow be mitigated.
One way to do this, as Russell proposes, is to add uncertainty to the robot’s notion of value. More specifically, the robot needs to maintain an uncertainty about the value of changing the world when it doesn’t have sufficient information about that value.
For example, if the robot makes coffee for its owner using that owner’s coffee pot, the world stays more or less the same, apart from the owner having her coffee. A bit of water and electricity have been used. A few dishes are dirty and there’s, of course, less coffee, but by and large, the world hasn’t changed very much. Typically, this is a good outcome: The biggest change in the world is the one you wanted, and the others are negligible. The robot couldn’t have acted in a way that result in negative consequences according to human values simply because the robot’s actions didn’t have that many consequences at all.
Contrast this with breaking into the coffee shop and having the police called. This represents a pretty significant change to the state the of the world, and this change was effected by the robot’s behavior. If the robot can see this coming (and if it’s smart enough, it can), then this should at least give it a reason to be suspicious that this might be a bad idea. If the robot doesn’t know if this is bad (relative to not having coffee), it might decide that the risk of it being bad is enough not to try it. Or, alternatively, the robot might ask someone for more information about the choice it’s thinking about making.
Said one way, the robot should strive not change things unless it understands the impact of that change. Said another way, we’d like the robot to have a sense of humility about its own ideas of what is best for the world at large. Sounds like a pretty good idea (and maybe not just for robots).
Provably Beneficial AI
These arguments all seem kind of squishy and non-technical, but the nice thing about them is that they can be formalized in actual concrete mathematical terms. Even better, once formalized, we can use this mathematics to prove that a certain behavior will be beneficial to its owner. That is, given some amount of information about human preferences, our robot can have strong guarantees that its behavior does not run contrary to the owner’s values.
Does this guarantee that all robots will always behave in a way that benefits all people? No. If a nefarious human gives a robot the idea that, say, compliance with the law has negative value, the robot will still act accordingly. Even if a well-meaning human correctly programs their values, there’s no guarantee the robot won’t behave in a way that runs contrary to the values of some other person. The guarantees here are that the robot will not violate preferences that the owner has that the robot has only limited information about, which is a much more difficult, common, and dangerous problem.
Since Russell started promoting this line of thought, a number of critics have pointed out that Russell is a person from a certain country, of a certain race and gender, with a certain religious inclination, and so on. Why should we trust him (or anyone) to put a system of values in place for our robot? It’s a sound argument, but it’s occasionally taken too far when these critics say that because no one can really be trusted, we should avoid Russell’s ideas, presumably to maintain some level of “neutrality” or that “it will do more harm than good to give a robot values”.
It’s important to realize that any software with a goal (which is really to say, all software) is neutral in absolutely no meaningful way. Anything that takes action in the world has an implicit set of values that are absurdly far from anything approaching what we would recognize as “good”. The coffee fetching robot already has values that, as we’ve seen, are completely broken in a very well-defined sense.
The question isn’t whether we should all buy into the value system of the programmer as being the right ones, the question is whether the programmer’s values are better than the ones you’d get otherwise. Most people would probably agree that any human could do better than “by all means, use lethal force to acquire coffee.” The former isn’t perfect, but if you had to choose, you’d probably take it over the latter.
Reasons for Optimism
I was on a panel recently with Jorie Koster-Hale discussing the changes to society that pervasive AI will bring. We were talking about employment, but she had a hopeful perspective that I think applies here as well. She pointed out that there aren’t many strong reasons to believe this change will be much worse than past societal changes, and there are reasons to believe that it will be better.
While powerful AI does have the potential to be a destructive force, people like Dr. Russell are out ahead of it. Neither panic, nor denial, nor sanguine acceptance will produce anything useful. But by being precise about the problems that AI might have and what their solutions might look like, we have a real shot at creating something that will transform our lives for the better.
Once upon a time in the Wild Wild West, a company with big aspirations was founded in a small town called Corvallis. If you’re thinking to yourself, “Wait, where is Corvallis?” don’t worry, you’re not alone. The BigML Team gets this question all the time when we present at conferences, events, and meetings around the world. We also get frequently asked, “How come BigML is not founded in the Silicon Valley?”
Here at BigML, we are proud of our small town roots in Corvallis, Oregon, USA. Corvallis may be little-known outside of the USA or even Oregon, but big things often come from small beginnings, as the saying goes. So, to answer these questions FAQs, we want to share the key reasons why BigML was founded in Corvallis, and why a big part of our team continues to love it there.
Silicon Valley Versus Silicon Forest
Many companies believe that to innovate, you need to have a direct presence in the Silicon Valley, and they dream of recreating a similar innovation engine elsewhere. However, in spite of Silicon Valley’s advantages like the strong density of experienced entrepreneurs and investors, these come along with a number of problems that limit innovation such as:
- High cost of living
- Talent inflation due to heavy competition
- Challenges of large cities (noisy, less safe, etc.)
It may come as a surprise to some, but the innovation rates per capita in the Silicon Valley are not that high when relatively compared to other, lesser-known innovation hubs in the USA like Corvallis. Corvallis, which resides in the Silicon Forest, has many advantages:
- Lower cost of living
- Smart and experienced workforce
- Culture of innovation
- High quality of life
Another plus to the workforce in Corvallis is the retention rates. People tend to be loyal and stay with companies for years, compared to the “mercenaries” in the Silicon Valley where people often jump from one job to another for a salary increase each year.
About Corvallis
Corvallis is consistently rated as one of the safest, smartest, greenest small cities in the USA. It’s located in the heart of the Willamette Valley in Oregon, USA, with a population of 55,000+. This does not including the approximate 25,000+ students attending Oregon State University (OSU), which is ranked among the top 1 percent of world universities.
Corvallis serves as a conducive place for innovation, for students and entrepreneurs alike. As an example of what makes this town special, Corvallis hosts da Vinci Days, an annual event that celebrates the community’s creative spark in Science, Technology, Engineering, Arts, and Mathematics (STEAM) fields. Who knew that you could bike on water?!
A Thriving Hub
Turns out, we aren’t the only ones who believe Corvallis is a great place to live and to build a company. But you don’t have to just take our word for it. Livability released a report in January listing Corvallis as the 3rd best place to live in the United States, and NerdWallet ranked Corvallis as #4 of America’s Most Innovative Tech Hubs, following San Jose, CA; Boulder, CO; and San Francisco, CA. Even better, Corvallis was ranked #1 for patents in the United States in 2011, which coincidentally was the year that Apple acquired 32 Patents, 5 of which were invented by Francisco Martín, BigML’s CEO, and others in Corvallis (e.g., System for browsing through a music catalog, Mediaset generation system).
BigML fits right in with this innovative community, with 6 patents and 12 patent applications of its own. Besides BigML, there are a number of leading companies that have set up shop in Corvallis, including both global enterprises and small startups. Here are just a few examples:
OSU Produces Talent Pool – Including BigMLers!
The strong, innovative community in Corvallis comes from its highly educated workforce. Over 50% of Corvallis adults have a bachelor’s degree or higher , and the city boasts a low unemployment rate of 2.7%, as of December 2017. Although Corvallis itself is not a large city, one million workers live within a 70 mile radius of Corvallis , so there is deep talent pool in the area. Oregon State University, and other schools in the region such as the University of Oregon (UO) and Portland State University are big drivers of this smart, experienced workforce. But make no mistake, BigML is loyal to the Beavers (OSU). For those outside of Oregon, there is a strong rivalry between the Beavers and the Ducks (UO).
When it comes to research, OSU leads the pack as the #1 public research institution in Oregon (receiving $441 million in research funding in 2017). On a national level, OSU’s College of Engineering is one of the largest and most productive engineering programs, including their leading Artificial Intelligence and Robotics program. Of particular interest is the Explainable Artificial Intelligence group, which received a $6.5 million DARPA grant to make Machine Learning based systems (e.g., autonomous vehicles and robots) more trustworthy by enabling these systems to effectively communicate with humans.
Seven BigML Team members studied at OSU (including the CEO, Francisco Martín), and they received a total of 10 degrees (3 Bachelors, 3 Honors Bachelors, 2 Masters, 1 Ph.D. and 1 Postdoc). BigML’s Chief Scientist, Tom Dietterich, is a Distinguished Professor (Emeritus) and Director of Intelligent Systems at OSU. Which brings us to the main reason why BigML was founded in Corvallis…
BigML’s Roots at OSU
Back in 2003, Francisco Martín moved to Oregon to work with Professor Dietterich on a number of DARPA-funded Machine Learning projects. While studying in Dietterich’s Machine Learning group at OSU, Francisco met Adam Ashenfelter, BigML’s Chief Data Engineer and Co-founder, and Charles Parker, BigML’s VP of Machine Learning Algorithms.
Fast forward to 2010. One thing leads to another, and Martín, Dietterich, and Ashenfelter start discussing a new venture that will build a cloud-based Machine Learning service…you guessed it, BigML! In January of 2011, BigML was founded with the mission of making Machine Learning beautifully simple for everyone. Later that year, Parker and Poul Petersen (another OSU grad) joined the team. So there you have it – how BigML began in charming Corvallis, Oregon.
Calling all Corvallis BigML Users
The BigML Team in Corvallis will be presenting at the upcoming Willamette Innovators Network (WiN) PubTalk event, which brings together the local business community to network and share ideas. If you are a BigML user in the Corvallis area, it’d be great to have you join us this Tuesday, February 13 from 6-8pm in Downtown Corvallis. Please see the WiN events page for more details.
Lastly, BigML loves to hire local talent, so check out our openings if you’re interested in bringing Machine Learning to the masses from Corvallis with the BigML Team!
Nowadays, our models deal with streams of information. They need to be updated when they are fed new data that differs significantly from the data used to train them. This is why Machine Learning tools need to provide not only accurate, but scalable, repeatable and ready-to-go solutions that can be swiftly integrated into any information system. This means that a Machine Learning tool shouldn’t necessarily look for the perfect immutable model, but rather for flexible solutions that can be retrained easily and brought to production in the shortest time as circumstances shift.
Today we’ll present a new tool that will help you better retrain your existing models as new data arrives. A new bigmler retrain sub-command has been added to BigMLer, the command-line to automate workflows in BigML.
Why and how to retrain?
The typical scenario to build a Machine Learning model involves some trial and error. This can be done manually or automatically, using scripts like best feature selection or SMACDown, but some tasks are meant to be repeated, for example: uploading your data, identifying the fields important to your model, evaluating the predictions or checking the model performance. Because your business or problem domain may change, your models will need to be retrained using new data eventually. But how do we do that?
Well, you need to realize that when you train a model in the first place, your data goes through some tasks: setting a type to each field, transforming the field values, adding new fields or filtering rows. The model rarely can work on the raw data, so to add more data to an existing model, all these transformations need to be re-run first.
In BigML, once a model (or any other resource) is built, it is immutable. None of its essential properties can be changed (only non-essential attributes, like their name or description, can be updated). The model keeps the information about what resources were used to build it or which configuration choices were made to get it. That ensures that all the steps that led to your actual model can be deduced i.e, the model-building workflow. By simply applying this workflow to new data you can get a retrained model.
Using BigMLer to retrain your models
Let’s assume you train a model from a sample of data and want to rebuild it with more data later. You can use a BigMLer command such as:
where you provide the file diabetes_sample.csv that contains the first sample of data and add a unique tag to the generated resources. Tags in BigML act like keywords that can be used in searches to retrieve your resources. In the command, we added a best_diabetes tag to our model so that we can use it as a reference when we decide to retrain it.
In that case, the call to retrain our model using BigMLer would be:
The process that BigMLer will execute for you consists of:
- Uploading the data in diabetes_new_data.csv to the platform.
- Retrieving the workflow that was used to build the last model with the tag best_diabetes.
- Adding the uploaded data to the dataset used in that model.
- Generating a new model from the merged data by using the same configuration arguments: the retrained model.
Every time you run a bigmler retrain command on a certain model, a new model resource is generated. BigMLer uses the directory set in --output-dir to store files that contain the information about the resources generated in this process. It also prints a report in the console showing the evolution of the process. The URL that will retrieve the latest retrained model can be seen at the bottom of this report. You can use this URL in any application that needs to use the newest model available to make predictions.
Following the example, you would use the
https://bigml.io/andromeda/model?$BIGML_AUTH;limit=1;full=yes;tags=best_diabetes
query to retrieve the latest retrained model. The $BIGML_AUTH environment variable stands for the credentials information (username=[my_username];api_key=[api_key]).
The output of this URL is a JSON with an objects attribute that contains a list with a single element: the model information. To use the latest retrained model in your information systems, you could pass this information to a local model object like the ones we offer in any of our bindings. The local model provides a function that generates the model’s predictions.
Retraining by windowing
Other scenarios involve retraining your models periodically with a subset of the most recent data. An example could be a sales model that will be updated monthly with a limit in the number of months of data to be considered. In this case, your data will need to be uploaded incrementally, but it shouldn’t be aggregated. Instead, we would need to keep monthly data in a separate dataset. We can easily achieve this by setting the number of datasets to be used in the model construction in the --window-size option:
This will now upload your data, build a new dataset with it and use a list of the last datasets specified in --window-size to rebuild an ensemble.
Referring by ID
Of course, the use of tags is just a human-friendly mechanism. It’s not strictly needed to add a tag to retrain a model. Since each resource in BigML has its own unique resource ID, you can use also that as a reference in the bigmler retrain command:
Consequently, the workflow that built the resource with the given ID (a cluster in this case) will be reproduced, so a new cluster will be built on the consolidated data.
As we’ve just explained, thanks to BigMLer, retraining your Machine Learning models is as simple as running a single command. This process can be triggered by any local scheduler available in your machines. To boot, you’ll be able to use the latest retrained model thanks to a permanent URL. A real simple setup for a production ready Machine Learning system. Your turn: just give it a try!
We are excited to introduce organizations, which makes it easy for companies to adopt Machine Learning across their entire corporate structure.
By creating an organization, the BigML Dashboard becomes a collaborative workspace where all the users in the organization can access, work on, and visualize the same projects and resources. Furthermore, organizations enable you to define different roles and permissions for each user involved on your Machine Learning projects, in a similar manner to GitHub repositories.
To show BigML organizations in action, we will cover a use case that is particularly relevant at this time of the year: gym membership churn. Many people start exercising in the new year, but their proposed health plans go awry after a few weeks or months, despite their best intentions.
Gyms offer enticing memberships to combat this tendency and encourage customers to make long-term commitments. Let’s see how a gym chain that we’ll call “Greg’s Gyms” can improve their membership retention rates by analyzing their customer data and sharing it with the right people within their company by using BigML organizations.
At Greg’s Gyms, the data analysis team is the primary user of BigML and they need to share models and resources amongst themselves, as each member focuses on a different part of the overall Machine Learning process. For example, one person takes care of the data preparation, one primarily builds the predictive models, and another person supervises the whole process to make sure everything is correct. Additionally, the data analysts want to share their insights with other departments that will benefit downstream, such as the customer success team. Now with the BigML organizations capability, these team members can work together on projects according to their assigned roles and permissions that regulate their actions on BigML. Let’s dive in and see how to set up your organization in four easy steps.
Create an Organization
The organizations feature is available for any user with a BigML account (whether it be a free account, a subscription, or a Private Deployment). However, to create an organization, you will need to purchase a BigML subscription that will be applied to your organization. From the BigML Dashboard, you can create an organization by clicking on the drop-down menu, as shown below.
Next, you will need to enter some basic information to sign up your organization. Greg’s Gyms needs to put the following details to get started:
- Organization Name: instead of “Greg’s Gyms”, we’ll set it to “gregs_gyms”. This name will be used in the URL of the organization page that lists all the projects, so it cannot include spaces or characters other than letters, numbers, underscores, and dashes. The “display name” can be adjusted later on, and can include spaces and other characters.
- Organization E-mail: since “Jones” is creating this organization, his e-mail is pre-filled by default. This e-mail is the one where BigML will send all the notifications regarding the organization’s activity.
- Billing Information: here Jones can add the billing details for Greg’s Gyms (tab highlighted in the screenshot below). This information is optional; Jones can include it now if desired, or he can add or edit it later.
- Subscription Plan: Jones will need to buy a subscription plan to create an organization. Since he wants other team members to enjoy this organization’s subscription, Jones will put 5 “additional users”. All organization members will enjoy the same subscription when they work in the organization.
Add Users and Assign Roles
Each user that is part of an organization on BigML will have an associated role, which governs their actions within the organization. Each role has different permissions, which regulate the user’s actions within projects (explained in the next section). There are four different roles: owner, admin, member, restricted member.
- The owner of the organization has full access to projects and the organization account and is the only one that can manage the subscription plan.
- The admins of an organization have full access to projects, can invite other users to the organization, and manage their permissions.
- Members can create new projects, and access public and private projects where they have at least read permissions.
- Restricted members can access public projects and private projects where they have at least read permissions.
For our example, Jones is the owner by default as the creator of the organization. This means he manages the subscription plan of the organization and also has “admin” privileges. Jones can now invite other BigML users (both internal and external to Greg’s Gyms) to join his organization with their usernames. Each user will receive an e-mail to confirm the invitation. At least one other user must be added to the organization besides the creator. The GIF below shows how to add users via the “organization settings”, inviting the user “gbrown” to join as a “restricted member” (click on the GIF for a larger view).
Individual users can belong to one or more organizations. As the different users work together in the organization, all the resources created will always belong to that organization (not to the user that creates them). For example, if Jones creates 5 datasets and 10 models in the “Greg’s Gym” organization, those 15 resources will all belong the organization and not his personal account. Jones can navigate between the different Dashboards for his personal account and his organization account(s) using the drop-down menu at the top left of the Dashboard, as shown below.
When Jones is working in his personal account (My Dashboard) he will enjoy his personal BigML subscription; when he is working in the “Greg’s Gyms” organization, he will enjoy the organization’s subscription. Resources can be transferred from his private Dashboard to an organization.
If a user’s account is deleted or he is removed from the organization, all the resources created by him in the organization will belong to the organization. If this were the case with Jones (who is the organization’s owner), another member would need to become the new owner.
Create Projects
After adding users and assigning roles, the next step to set up an organization on BigML involves projects and user permissions. All the resources in an organization need to be created within a project.
Continuing with our example, Jones now needs to create a project for his organization members to work on. Since the company’s current goal is to increase the number of customers who commit to year-long gym memberships, he creates a project titled “Gym Membership Retention”. Below we show how to toggle between different projects by using the drop-down menu at the center of the Dashboard, how to create a new project, and how to search in the project list view (click on the GIF for a larger view).
When creating a project, you must add a project name, and set it as public or private.
- Public projects allow all users in the organization to access the projects with write permissions.
- Private projects only allow members invited to the project to have access.
Additionally, you can add tags and a description for your project, or you can add these details later. After saving your new project, you will land in the project list view, where you can see all of the organization’s projects. Here you can search for projects by name, and see an overview of all of the resources created within a project.
Assign User Permissions
Now that he has created a project, Jones can assign users different permission levels for that project. There are three levels of permissions per project: manage, write, read.
- Manage permissions in a project gives you full access to the project resources and allow you to invite other users and edit their permissions within the project.
- Write permissions allow users to create new resources in a project.
- Read permissions allow users to view the resources in a project.
Assigning different permissions allows the organization’s admins to make sure the right people within an organization have the right access to resources on BigML. For example, Jones may want to give write permissions to his team of data analysts for their “Gym Membership Retention” project, but only give read permissions to the customer success team, since those employees will not be building models and making predictions. But by giving their customer success team access to view the models and predictions, this enables them to bring their unique domain knowledge to the table, creating an effective “meeting of the minds” to help solve the company’s customer retention challenges.
So there you have the basics of creating an organization on BigML! In our ongoing effort to make Machine Learning accessible for everyone, the BigML organizations feature is a big step forward to make company-wide adoption of Machine Learning easier than ever before. For more details about BigML organizations, please see our documentation on the dedicated release page. Try it out, and let us know your feedback and any questions at support@bigml.com.
Want to know more about Organizations?
If you have any questions or you’d like to learn more about how organizations work, please visit the dedicated release page. It includes the BigML Dashboard documentation, the webinar slideshow as well as the full webinar recording.
To wrap up this series of blog posts, we’re going to get a little bit deeper into the business of setting operating thresholds. The technical details of applying an operating threshold are fairly simple: you choose a class and a threshold, use BigML to make a prediction for some instance, and apply the class if the prediction’s probability is over that threshold. As we’ve seen, this is easy to do via the Dashboard, the API, and with WhizzML and the Python Bindings.
What’s a little more tricky to pin down technically is how you set this threshold for your particular data. One of the dirty little secrets of Machine Learning is that this almost always needs to be done, and Machine Learning can’t do it automatically. This is because the algorithm can’t know from your data alone which sort of mistakes are tolerable and which are unacceptable. It needs your help!
Thankfully, we’ve developed a tool that takes a lot of the pain out of discovering where the best threshold is. When you create a BigML evaluation, the information you need to set for this threshold is at your fingertips. Let’s dive in.
Splash
As we mentioned in a previous post, you create a BigML evaluation by first creating a training and test split, then training a model on the former, and evaluating it on the latter. By default, the evaluation view is a collection of numbers that summarize the performance of your model with the default threshold.
However, if you click on the ROC curve button at the top, you see the ROC curve, a selector for the positive class, and a slider to select the probability threshold. You also see a whole bunch of numbers next to the curve, which we won’t go into here and for the purposes of threshold-setting usually aren’t necessary. Our business here is mostly with the curve and slider.
The curve you’re seeing is the ROC curve (Receiver Operating Characteristic) of this model’s predictions with regard to a particular class (which you can set in the “positive class” dropdown above the curve). On the y-axis is the true positive rate and x-axis is the false positive rate. Each point on the curve represents a threshold, with its own true positive rate and false positive rate. At high thresholds, both numbers will be lower (you’re labeling less examples positively because you’re setting the probability bar high). As you lower the threshold, your true positive rate will go up, as you pick up more of the positive examples (good!), but so too will the false positive rate (bad!). The threshold setting game is to pick up as many true positives as you can while only picking up as many false positives as you can tolerate.
As a modeler, what you’d like is for this curve to go up as fast as possible on the left side of the plot (indicating that the examples labeled with the highest probability for that class actually are examples of that class), then flatten out as you move right. The best possible model will have a curve that is a right angle at the upper left of the plot, and a model that’s no better than random will be a diagonal line through the center of the plot. The closer you get to that right angle, the better of a model you have.
Is it safe?
One nice thing you can see from a curve like this is how reliable threshold-setting will be for this model. Because the data you apply the model to will be slightly different from your test data, your chances of not getting exactly the performance you see for some threshold are pretty high.
If the curve is fairly smooth, like the one above, this isn’t a big problem: it means the performance of one threshold is much like the performance of another. If it isn’t, it means that adjacent thresholds appear to have radically different performances, and so your performance in the “real-world” might not even be close to the performance you see in the confusion matrix for the given threshold.
What do you do if you see this sort of curve? This basically indicates that you need more data. My own experience is that you’d prefer to have at least a few hundred instances in your evaluation dataset to conduct an accurate evaluation, and anything less than a hundred starts to be a bit scary from an evaluation-certainty perspective.
The Devil and the Deep Blue Sea
We’ve gone through a few threshold setting use cases in previous posts, and the way you set the threshold typically involves sliding around the threshold slider until you see a confusion matrix that you like. This is a good way to do it, but it typically involves having both a very good idea of exactly the costs of mistakes in your domain, as well as potentially looking at every threshold. Are there some rules of thumb in cases where you lack this certainty or patience? Thankfully, there are a few things you can try right off the bat:
First, look at the confusion matrix for the default threshold of 50%. If it’s satisfying, you don’t have to do anything, which is always best.
Next, on the slider, you will see a vertical mark labeled “Max. phi”.
This is the threshold that maximizes the phi coefficient for this class. Phi is a measure that accounts for class imbalance in the results, so if your positive class is rare but important to get right (think: medical diagnosis), then this might be a better choice. Try sliding the threshold selector over the vertical line and see if your confusion matrix is more to your liking.
Click on the plot for the K-S statistic:
This is another way of showing, as you lower the threshold, how fast you pick up the positive versus the negative examples (the former is the blue curve and the latter is the black curve). You’d like a threshold where the distances between the curves are as large as possible, indicating that you’ve picked up a high percentage of the positive examples and a low percentage of the negative ones. We’ve drawn a dotted line on the plot where this distance is largest.
Move the slider until your threshold matches that dotted line and check out the confusion matrix at that point to see if it’s better suited to your context.
Wait! There is . . . More?
Oh, you didn’t know about the other curves? For experts and intrepid beginners, we also plot the precision-recall and lift curves, which give you still more information about your model’s performance. You can read more about these other curves on THE INTERNETS, or in our own documentation.
Want to know more about operating thresholds?
If you have any questions or you’d like to learn more about how operating thresholds work, please visit the dedicated release page. It includes a series of six blog posts about operating thresholds, the BigML Dashboard and API documentation, the webinar slideshow as well as the full webinar recording.
This blog post, the fifth of our series of posts about operating thresholds, focuses on two ways to automate the use of operating thresholds in our predictions (single or batch) and evaluations. The first way involves WhizzML, BigML’s Domain Specific Language for Machine Learning workflow automation. It allows you to execute complex tasks that are computed completely on the server side (with built-in parallelization), where all resources involved are treated as first-class citizens by BigML.
The second way to automate operating thresholds involves the set of bindings that BigML maintains, which allow developers to work in their favorite programming language as they interact with the BigML platform. The Python binding is already updated to handle the most recent platform capabilities we’re covering so we’ll use it as an example in this post. However, there are other bindings options such as Java, C# or Node.js.
Operating thresholds can be used with any of your classification models: decision tree models, ensembles, logistic regressions, and deepnets (neural networks). They can improve the quality of your single and batch predictions, and thus, your evaluations.
Let’s see how to automate operating thresholds for a single model by using WhizzML. In our example, we’ll deal with a classifier that has two possible outputs: good or bad. We’re interested in improving the classification of the instances that fall into the good category, so we’ll set this one as the positive class. To ensure that we reduce misclassifications, we’ll consider that our model predicts good only if the probability of this prediction goes over a 60% threshold. To see whether this improves the model’s performance, we can create an evaluation via WhizzML.
;; creates an evaluation setting an operating point ;; for a single model (define my-evaluation (create-evaluation { "model" my-model "dataset" test-dataset "operating_point" { "kind" "probability" "positive_class" "good" "threshold" 0.6 }}))
Remember that almost all BigML resources (except predictions and projects) are asynchronous, so if you want to use the evaluation my-evaluation or its components, like the accuracy, precision, etc., you will need to ensure that the creation process has actually finished. That’s what the next snippet of code does.
;; creates an evaluation with an operating point ;; for a deepnet and retrieve its precision (define my-eval-precision (get-in (create-and-wait-evaluation { "deepnet" my-deepnet "dataset" test-dataset "operating_point" { "kind" "probability" "positive_class" "good" "threshold" 0.6 }}) ["result" "model" "precision"]))
If you prefer the BigML Python bindings, the equivalent code is:
from bigml.api import BigML api = BigML() args = { "operating_point": { "kind": "probability", "positive_class": "good", "threshold": 0.6 } } my_evaluation = api.create_evaluation( "deepnet/59b0f8c7b95b392f12000003", "dataset/59b0f8c7b95b392f12000001", args) api.ok(my_evaluation) precision = my_evaluation["object"]["result"]["model"]["precision"]
For more details about these and other evaluation properties, please check the dedicated API documentation.
Single predictions can also be computed by using this operational threshold. In fact, these evaluations are computed by averaging the matches of single predictions whose outputs are already known. Let’s see how to set a confidence threshold of 0.66 for a prediction made with a logistic regression by using WhizzML.
;; creates a single prediction setting a confidence threshold (define my-prediction (create-prediction { "logisticregression" my-logistic "input_data" {"000001" 0.35,"000002" 1.2} "operating_point" { "kind" "confidence" "positive_class" "good" "threshold" 0.66 }}))
These plus other prediction properties are also explained in the API documentation. Using the BigML Python bindings, the equivalent code would be as follows:
from bigml.api import BigML api = BigML() args = { "operating_point": { "kind": "confidence", "positive_class": "good", "threshold": 0.66 } } input_data = {"000001": 0.35,"000002": 1.2} logistic_id = "logisticregression/59b0f8c7b95b392f12000000" my_prediction = api.create_prediction(logistic_id, input_data, args)
As we mentioned above, you can also use operating thresholds for your batch predictions. Below you can see how to create a batch prediction in WhizzML by using an ensemble of ten models, where we specify that at least six models should return the positive class in their predictions.
;; creates a batch prediction setting ;; a threshold based on the number of votes (define my-batchprediction (create-batchprediction { "ensemble" my_ensemble "dataset" my_dataset "operating_point" { "kind" "votes" "positive_class" "good" "threshold" 6 }}))
With the BigML Python bindings, you should write this code as:
from bigml.api import BigML api = BigML() args = { "operating_point": { "kind": "votes", "positive_class": "good", "threshold": 6 } } ensemble = "ensemble/59b0f8c7b95b392f12000001" dataset = "dataset/59b0f8c7b95b392f12000003" my_batch_prediction = api.create_batch_prediction(ensemble, dataset, args)
Easy, right? If it’s your first time using our Domain Specific Language for Machine Learning we recommend that you check this series of blog posts to get familiarized with WhizzML.
Want to know more about operating thresholds?
If you have any questions or you’d like to learn more about how operating thresholds work, please visit the dedicated release page. It includes a series of six blog posts about operating thresholds, the BigML Dashboard and API documentation, the webinar slideshow as well as the full webinar recording.
