Here’s the first one, that shows how to go from data concept to BigML dataset.  We hope you find it useful in your own machine learning explorations!

]]> http://blog.bigml.com/2013/06/17/giving-your-data-the-machine-learning-treatment/feed/ 0 charleslparker charlie_parker http://blog.bigml.com/2013/06/13/matter-over-mind-in-machine-learning/ http://blog.bigml.com/2013/06/13/matter-over-mind-in-machine-learning/#comments Thu, 13 Jun 2013 17:39:58 +0000 charleslparker http://blog.bigml.com/?p=4486 ]]> I am fortunate enough to have had a number of conversations with Dr. Kiri Wagstaff of NASA’s JPL on a number of occasions (you might as well get the jokes about “not having to be a rocket scientist to understand machine learning” out of the way right now).

Wagstaff is a brilliant scientist.  On top of that, and fortunately for all of us, she works very “close to the data”, using her machine learning expertise to solve important problems that directly impact people other than machine learning researchers.  This closeness to the data is somewhat rare among machine learning experts but is becoming more and more common.  Our own chief scientist Tom Dietterich is a pioneer in computation sustainability and ecosystem informatics.  Another acquaintance of mine, Rayid Ghani, left a lucrative position at Accenture Research to head up President Obama’s vaunted data analytics team.

At the 2012 ICML conference, Wagstaff wrote an excellent little position paper calling for “Machine Learning that Matters” (besides the paper, there’s also an online forum).  In it, she points out (with little argument from the gallery) that machine learning papers typically proceed as follows:  Find a dataset for a problem that is only partially solved, invent or improve an algorithm to solve this problem such that some improvement on some standard metric is made on that dataset, write it up and publish.  Repeat until tenured.

Wagstaff notes in her paper that, while this approach produces a substantial number of computer science professors, it isn’t nearly as successful at getting machine learning algorithms in a position to make a measurable difference in society at large.  At fault, she says, is both the data we use to test algorithms and the metrics we use to evaluate our tests.  Many of the datasets we use in the scientific literature in machine learning have been around for years or even decades.  Producing a small performance improvement on the vast majority of these datasets will have little effect outside of machine learning conferences.  And what do we even mean when we say “small performance improvement”?  If the AUC of my classifier on some dataset is 0.85 and that of yours is 0.81, what does that say about the consequences of using my classifier versus yours in the context of that data?  The answer, of course, is totally dependent on the data.  And if that data is meaningless, then we’re well on our way to producing technology useful to absolutely no one.

She then goes on to list some of the challenges to socially relevant machine learning, and calls for a renewed focus on real-world data in the machine learning literature.  She also calls for metrics that reflect the real world impact of the technology (think, instead of “F-Measure”, of something like “gallons of water sanitized” or “doses of vaccine distributed”). Finally, she provides a very nice list of challenge problems for machine learning researchers, which we’ll get to later.  Definitely read the whole paper, which is very accessible to non-experts, to get the full-fat version of the argument.  For now, though, I’m going to play devil’s advocate a bit with Wagstaff’s excellent work.  I’m not doing this to be argumentative, as I mostly agree with everything she is saying.  Rather, I’m going to use it as a jumping off point to maybe inspire some of the work she’s hoping will happen.

Sympathy for the Devil

The conundrum that Wagstaff points out is possibly as old as science itself. Dijkstra advises scientists to strive in their work for both “scientific soundness” and “social relevance”.  Science, especially computer science, occupies a shadowy space between mathematics and engineering.  Like mathematicians, we would like our work to be abstract and general, to say something fundamental about optimization or, even better, about intelligence. Like engineers, however, we would also like to solve a specific problem in the world.  To be able to point to a cured patient, or an organized news feed, or a translated document and say “my algorithm did that”.

And therein lies the problem:  The more abstract and general something is, the less specific it is. Machine learning scientists, both writers and reviewers, faced with the choice, tend towards the former. Part of the reason, I suspect, is that it’s easier to retain some degree of scientific objectivity when dealing with the abstract.  Suppose a paper proposes a classifier system that lowers the total cost of cancer screenings by half and only raises the rate of undetected cancers by 1%.  One suspects that reviewers who have lost loved ones to cancer may feel differently about this than ones who have not. Another part of the reason is a healthy suspicion of solutions that don’t generalize past a single case, as such niche solutions are an important step on the road to charlatanry.

Nonetheless, science can and does impact the real world in very specific ways. The frustrating part about machine learning in particular is that we are so tantalizingly close to having our cake and eating it, too.  A century ago, Einstein labored at relativity with (probably) no thought to GPS satellites, so he could not have appreciated the full effect his work would have on society. Scientists are accustomed to the idea that our work might not be useful to society for years after it occurs, and so there’s no stigma attached to producing work that is scientifically sound but not (yet) socially relevant.  But machine learning is so directly applicable to so many problems that at times it seems downright easy to find a landing zone for these algorithms. Yet, the traditional scientific mindset pulls us back to the abstract.  Even though the gap between theory and practice in machine learning seems nearly bridged, it looks like we still need a little help.

Fixing a Hole

In Wagstaff’s paper, she lays out a program of challenges for machine learning researchers interested in making a difference in the real world:

1. A law passed or legal decision made that relies on the result of an ML analysis.
2. $100M saved through improved decision making provided by an ML system.
3. A conflict between nations averted through high quality translation provided by an ML system.
4. A 50% reduction in cybersecurity break-ins through ML defenses.
5. A human life saved through a diagnosis or intervention recommended by an ML system.
6. Improvement of 10% in one country’s Human Development Index (HDI) attributable to an ML system

This is an excellent set of challenges, and I would guess that few or none will be solved by traditional machine learning scientists (with the exception of #5, which I think may have already been met depending on how one defines “saved”).  This isn’t because the problems themselves are terribly hard from a machine learning perspective (though they might be), but simply because we don’t have many machine learning researchers at high levels of government, or writing laws, or responsible for network security.  They have spent their careers studying machine learning, in all of its arcane mathematical glory.  To think they might also know enough about law to apply it in that context is asking a lot.  Who will come to their aid?

You.

At BigML, we’ve committed ourselves to putting machine learning technology into the hands of people who can meet the challenges that Wagstaff has put forth. You have the data and the expertise to know which problems in your field of interest can be solved by machine learning; we have the machine learning expertise to provide you with that solution.  I don’t think we need machine learning gurus to tackle Wagstaff’s problems, I think we need gurus from other fields who know enough about machine learning to use it. Wagstaff alludes to this in her paper. If we can get usable, flexible, dependable machine learning software into the hands of domain experts, benefits to society are bound to follow.

If I Had a Hammer

I often tell people that machine learning is a hammer.  Machine learning scientists are busy every year, every day, improving that hammer, making it more durable, harder, improving the shape, and on and on.  But as Wagstaff points out, that hammer is no good to anyone until someone starts hitting nails with it.  BigML is giving the world a chance to swing that hammer, whether you’re an expert in machine learning or anything else.

Most of Quandl’s datasets are univariate, which provide interesting insight and lend themselves to interesting time-series forecasting models.  In addition, there’s a great utility in which you can combine columns from different datasets to create a more complex item called a Superset. BigML works very well with these multivariate Supersets.

You can choose several columns from different datasets, and build a custom Superset.  If the frequencies in your Superset are different (e.g., if one column tracks data monthly, while the other does daily) the Superset will normalize your data, adjusting to the lowest frequency (e.g., monthly instead of daily).

Creating a Superset

1. Locate the different datasets that contain the columns you want to add to your Superset:

2. Right click on the column header, and add the column to an existing Superset, or create a new one when adding the first field.

3. Be sure to check the fields’ frequency, because the final Superset frequency will be the lowest frequency of all the fields added.

See below a screenshot from a sample Superset, created from several datasets about economic and demographic indicators in the United States.

Quandl Superset

Use your Superset in BigML

Once the Superset is created and columns are renamed with the desired labels, we can proceed in one of four different ways to upload the result to BigML:

1. Click on Download button in Quandl and export as .CSV. This file is now ready to be uploaded to BigML as new Source. (For details on how to upload files into BigML, please visit here, and follow the links to our helpful videos)
2. Get the direct link, by clicking the “show API call” in the Download modal window within Quandl, below the Download Data button.Then use this link as external source in BigML:
3. Or use the Quandl link to upload the source using our public API:
   

   curl --silent https://bigml.io/source?$BIGML_AUTH \
        -X "POST" \
        -H "content-type: application/json" \
        -d '{"remote": "http://www.quandl.com/api/v1/datasets/USER_1O2/1O6.csv", "name": "USA Unemployment Rate"}'
   

4. Or you can upload the Source, generate the Dataset and create the Model directly with a single bigmler call (be sure to put the objective field at the end): 

   bigmler --train "http://www.quandl.com/api/v1/datasets/USER_1O2/1O6.csv" \
           --name "USA Unemployment Rate" --tag "US Indicators"
   [2013-05-25 03:29:11] Creating source.
   [2013-05-25 03:29:22] Source created: https://bigml.com/dashboard/source/51a013e9925ded36f4000103
   [2013-05-25 03:29:22] Creating dataset.
   [2013-05-25 03:29:25] Dataset created: https://bigml.com/dashboard/dataset/51a013f1925ded36f3000310
   [2013-05-25 03:29:25] Creating model.
   [2013-05-25 03:29:31] Model created: https://bigml.com/dashboard/model/51a013f5925ded36f3000314.
   
   Generated files:
   
   SatMay2513_032911
   ├─bigmler_sessions
   ├─dataset
   ├─models
   └─source
   

And finally the model is created in BigML.  In the view below the decision tree visualization has the unemployment rate as the predicted field.  In this case, unemployment is predicted at 5.97%, based on the fact that there are less than 14,549,000 government employees, the civilian employment ratio is between 58.75%-61.83%, population is less than 226,954,000, the US Dollar index is lower than 98.69, personal income is less than $2.192B, the average new house prices is greater than $62,950, the month is later than July, and monthly new vehicle sales are less than $14.92M.

You can also analyze your model using BigML’s sunburst visualization—which has three viewing options:  Split Field, Prediction and Confidence. Shown below we see the same finding (from the decision tree above) in the Prediction view, where darker means a higher result.

We’re just getting started in this collaboration with Quandl—stay tuned for more updates and innovations!  In the interim, if you build a model with a Quandl superset, please let us know.

Barcelona

On May 30, 3:00PM-5:00PM, BigML’s CTO and Hacker at Large jao will give a guest lecture on “Learning from Big Data Using the Cloud” to students at the Universitat Politecnica de Catalunya (located in North Campus, Building 204). This lecture is also free to attend.

San Francisco Bay Area

On June 3, 6:30-8:30PM, BigML’s CEO Dr. Francisco J Martin will speak on “A Few Challenges to Make Machine Learning Easy” at a meeting hosted by the ACM San Francisco Bay Area Chapter.  The meeting will take place at the eBay Whitman Campus and is free to attend.

London

On June 4, BigML’s VP of Corporate Development Andrew Shikiar will present as part of the AGC Partners’ European Growth Conference.  You can request an invitation from AGC Partners and/or contact us.

Chicago

BigML will have representatives attending Predictive Analytics World on June 11-12. If you’re attending and want to meet up please let us know!

We believe that one of the keys to fully democratize machine learning is making it very affordable. Many current machine learning solutions are not only costly in terms of complexity, time-to-productivity, configuration, installation, and hardware but also in terms of the licensing costs themselves. We’ve priced BigML using an affordable pre-paid model: with no licensing fees, no monthly fees, and we’ve even thrown in a free development mode. You just pre-pay credits that you can consume as you go. However, we have seen that when users become more familiar with BigML they want to use it to solve many other data-driven tasks, keep their models up-to-date, and create ensembles that may have hundreds of models.

For these users, we’ve been working on a new subscription model that will allow our customers to use BigML as much as they want for a monthly, quarterly, or yearly fee. The only limitations will be the maximum size of tasks and the number of parallel tasks. Customers will be able to select a specific plan or request a personalized one if their needs are different.

BigML subscription plans will be ready very soon. Actually, they are ready but we are still deciding what the right max size, max parallel tasks, and price for each different plan might be. As such, we would like to get  a few users to try our subscription plans for free before setting up the initial parameters. We thought a great way to decide which users should get free subscriptions was by getting them to invite some other friends.

If you invite three friends that sign up for free BigML accounts, you’ll get a free month subscription. With a free subscription you will be able to run unlimited tasks to create unlimited resources (datasets, models, ensembles, evaluations, and predictions). We’ll be limiting size and number of parallel tasks but don’t worry if you feel that our initial limitations are too strong for your current needs we’ll be happy to upgrade them. You’ll be able to get up to a 3-month free subscription plan if you manage to get nine friends to sign up.

In this post we introduce the three latest additions to BigMLer: dataset splitting, cross-validation and ensembles.

Datasets split into datasets:

As we saw in previous BigMLer’s blog posts, the --sample-rate flag lets you control the fraction of data that will be used to build models and, consequently, what percentage is left for evaluations. However, to have better control of your test data you may want to permanently separate a group of instances from your dataset. OK, says BigMLer, let’s split your dataset:

bigmler --train data/iris.csv --test-split 0.2 --evaluate

in a single line you will

• create a dataset with the entire data
• split the dataset into a test one, holding 20% of the instances, and a training one with the remaining 80%
• create a model with the training dataset
• evaluate this model with the test dataset

Your training and test data will then be accessible as regular dataset objects that can be recovered independently or even split again. If you have a look at the command’s output, you’ll see dataset_train and dataset_test files, which contain the ids for the generated splits. Let’s say that your training dataset is dataset/5188ddfa37203f085c000008 and your test dataset is
dataset/5188ddfd37203f085c00000b. Then if you want to try the same data but in a model built with statistical pruning:

bigmler --dataset dataset/5188ddfa37203f085c000008 \
        --pruning statistical

will build the new model model/518954b437203f1a6f000000 and

bigmler --dataset dataset/5188ddfd37203f085c00000b \
        --model model/518954b437203f1a6f000000 --evaluate

will use the same test dataset for the evaluation. Then you’ll be able to compare your models and choose the one that performs best.

While splitting a dataset once for evaluating a model is good, sometimes you want to create a handful of different splits to avoid basing your conclusions on a biased sample. Here again, BigMLer can help.

Tuning your models: cross-validation

Usually, when you build a model, you can adjust some parameters to improve the model’s performance. For instance, BigML lets you choose between several pruning modes. Each choice will give you a different decision tree, but how do you know which one works better for your data? Cross-validation will help you with that. As you may know, cross-validation is a technique that helps you estimate the validity of your model using samples of your training data. Nowadays, several types of cross-validation are normally used, but all of them proceed by building many models with samples of your training data and testing them with the data held out during the sampling.

In BigMLer we’ve chosen a Monte-Carlo variant of the cross-validation algorithm, which repeatedly splits at random the dataset into training and test subsets, builds the corresponding models and evaluates them, averaging the results. In this kind of algorithm, there’s no a priori relation between the size of the sample you use to evaluate and the number of evaluations you can run to validate (besides the fact that they should be enough to ensure good coverage of your data).  Never mind, you just have to say:

bigmler --train data/iris.csv --cross-validation-rate 0.1

and the tool will take care of the job, namely:

• create a dataset with all of the training data
• hold out a random sample of 10% of the dataset to run evaluations
• use the remaining 90% to build a partial model
• evaluate the model with the held out data
• repeat the previous steps using a different random sampling 2 * n times, where n is the percentage of held out data (in this case, 20 runs), to reduce variance
• finally, average all the partial model’s evaluations to get a close estimation of the complete model’s performance

Or, if you want to choose the number of evaluations, use the --number-of-evaluations flag:

bigmler --train data/iris.csv --cross-validation-rate 0.1 \
        --number-of-evaluations 20

BigMLer will store the results in an output directory (see BigMLer’s last blog post for details) where you will find cross_validation.json and cross_validation.txt files, which will contain the average of all the models’ evaluations.

Of course,  sometimes a single model can fail to perform well for your data. Again, BigMLer has a solution for you.

Models working together: ensembles

Ensembles are groups of models built by sampling a single dataset. Thus, the models in an ensemble are all different but similar enough, as they are built on a common part of information. As we saw in detail in previous posts, ensembles’ predictions are usually more accurate than those from a single model because the ensemble’s diversity helps smooth out small variations in the individual models, reinforcing their shared features. If you’ve read our first BigMLer post you’ll be familiar with ensembles, as they were available in BigMLer long before they made their appearance in the new version of BigML’s python bindings. Nevertheless, now that ensembles are first class citizens  in the BigML API, BigMLer has adapted to handle them as one of your regular resources.

For example, say you created ensemble/51630c4e37203f2292000082 and want to use it to generate your predictions locally. Just let BigMLer do it:

bigmler --ensemble ensemble/51630c4e37203f2292000082 \
        --test data/test_iris.csv

Using this command, the ensemble information is downloaded and the model predictions are combined into a final prediction stored in predictions.csv.

Similarly, evaluating the ensemble with test data amounts to:

bigmler --ensemble ensemble/51630c4e37203f2292000082 \
        --test data/iris.csv --evaluate

and you’ll find the usual evaluation.json and evaluation.txt in the output directory. How does that sound to you?

To sum up, BigMLer now includes new features like dataset splitting, model’s cross-validation and ensembles’ predictions and evaluations to help you get the best of your data. Want something else? Let us know and stay tuned!

A traditional approach is to split the available data into two disjoint sets, one for training and one for tests. If your programming skills are up to the task you could use, say, Weka, Scikit, R, or SQLS to perform the split. But, as we explain in this post, BigML offers you a much easier way to automatically perform this task with just one click or, if you use our API, a couple of HTTP requests.

On the surface, splitting the instances of a dataset into training and test sets seems like an easy task, for example just take the first 80% as the training set and the rest as the test set. But what if the dataset is ordered according to non-random criteria? Let’s imagine that the instances are ordered by the field representing the class that you want to predict.  Then, almost certainly, the test set will not be a fair representation of the instances in the training set, or the new ones in production for that matter. For example, say that you have an equally balanced dataset with 5 classes, ordered by class.  Then, taking the first 80% of the dataset instances as your training set, your test set will contain instances of just one class and the training set only instances of the other four classes. So you’d be testing a model with instances of a class that it never saw during training and therefore the performance measures won’t be representative of the real performance of the model. This shows that random sampling is important when splitting your data into training and test sets.

When it comes to random sampling, you could use the Clojure library that we open-sourced a few months ago, but if you don’t know Clojure there’s no need to worry: in BigML’s interface splitting a dataset into a training and test set is now one click away.

Now, in your dashboard, from the dataset listings or from an individual dataset view you have a new menu option to create a training and test set in only one click.  By default, the split for training and test set is 80/20. That is, 80% of the instances in the dataset will be used to create a new dataset suffixed with “training” and the other 20% of the instances will be used to create a new dataset suffixed with “test”.

To get the same training/test set split from BigML’s API two requests are needed.  Both requests need to share exactly the same following arguments:

• origin_dataset:  this is a newly created argument that specifies the dataset to split.
• seed: (e.g, “my seed”)  the seed to initialize the random number generator used to shuffle the rows.  It can be any string.  The same seed always gives raise to the same shuffling.
• sample_rate: a number between 0 and 1 that represents the fraction of instances. (e.g., 0.7 for a 70/30 split).

The first request is to create the training set.  We have updated the dataset creation REST method to accept a new argument named origin_dataset that specifies the dataset to split. You also need to specify the sample_rate that to use (e.g., 0.8) and a seed (e.g, “my seed”).  The seed is essential to guarantee that in the second call the random generator is initialized to exactly the same state, ensuring that the training and test sets are complementary.

curl "https://bigml.io/dataset?$BIGML_AUTH" \
     -X POST \
     -H "content-type: application/json" \
     -d '{"origin_dataset": "dataset/518d0568925ded7ea40004a2",
          "sample_rate": 0.8,
          "seed": "my seed",
          "name": "Training set"}'

The second request is to generate the test set. Notice that the sample_rate needs to be the same as in the training set creation (i.e., 0.8 and not  0.2 ). The key difference is the flag out_of_bag that needs to be set to true. This will create a new dataset with all the remaining (out-of-bag) instances that weren’t used in the training set creation.

curl "https://bigml.io/dataset?$BIGML_AUTH" \
     -X POST \
     -H "content-type: application/json"  \
     -d '{"origin_dataset": "dataset/518d0568925ded7ea40004a2",
          "sample_rate": 0.8,
          "seed": "my seed",
          "out_of_bag": true,
          "name": "Test set"}'

Once you have your training and test sets ready, you can use the training set to create a model or ensemble and the test set to evaluate it. If you are satisfied with the results, then remember to use the original dataset (the one with all your data) to create the model or ensemble that you’ll release to production.  In our next post, we’ll show you how to split a dataset from BigML’s command line.

I work at a consumer web company, and recently used BigML to understand what drives return visits to our site. I followed Standard Operating Procedure for data mining, sampling a group of users, dividing them into two classes, and creating several features that I hoped would be useful in predicting these classes. I then fed this training data to BigML, which quickly and obediently produced a decision tree:

Next I used BigML’s interface to examine the tree’s many subsets, shown as “nodes” in the diagram above. I moused over a node at the top of the tree and saw that it achieved high separation for a large fraction of the training set:

This one node covered 58% of the data, and separated the two classes with 73% confidence. (“Confidence” is a measure of node purity, and for this node 73% of the data belongs to class “0″.) With a little more work, I found another node that covered another 22% of the data, this time predicting class “1″. For the remaining 20% of data, the best rule I could find (after much mousing-over in the tree) was a single node with a lousy 51% confidence—barely an improvement over flipping a coin. I affectionately named these nodes Rule 1, Rule 2 and Blind Spot.

This is a common use for decision trees: gaining insight by finding the “best” nodes as measured by the fraction of data they cover (“support”) and their purity (“confidence”). When exploring a decision tree for insight, the goal is to find the smallest collection of useful rules that accurately summarizes your data.

Today BigML is launching the SunBurst visualization, which makes it much easier to gain insight from decision trees. Below is a SunBurst viz of my tree: the nodes are now shown as arcs, with the number of radians representing support and the color representing confidence. With a minimum of manual searching, I can easily find Rule 1, Rule 2 and especially the Blind Spot (which, together with its subsets, stands out in ugly, non-predictive brown):

Let us ponder the amazing feat this Burst of Sun has achieved. In an eight-dimensional data set of 48,000 instances, I can see immediately which nodes have the highest combination of support and confidence. But wait, there’s more: I can also see exactly how all of the nodes fit together in a tree hierarchy, which gives me further insight into the data. For example, the upper right of the tree shows several large subsets of Rule 1 that glow bright green, and a closer look reveals that these subsets stack like Russian nesting dolls, each one prettier (but smaller) than its parent. So if I felt that Rule 1 misclassified too many instances, I could easily select one of its prettier children instead, choosing higher confidence at the cost of lower support.

Try out the SunBurst confidence visualization for yourself by training a model, going to the Models tab, and clicking the hypnotic SunBurst icon:

Then just click the Confidence icon. May all your nodes be green!

Better visualization obviously does not solve everything, and the usual cautions about understanding your data and validating your model still apply. While the above model is useful (since I haven’t said anything about the actual data it’s trained on, you’ll have to take my word for it), I still want to try a larger data set, and validate the resulting model by splitting the data into training and test. Perhaps most importantly, we cannot assume that two subsets with similar confidence but different predicted classes (like Rule 1 and Rule 2) will make equally valid rules in practice, since a false positive could be much, much worse than a false negative, or vice versa.

Nonetheless, the SunBurst is a huge leap forward in decision tree visualization. BigML’s Adam Ashenfelter explains that the SunBurst “may not be as intuitive as our regular tree view”, and he’s right—it’s about a thousand times more intuitive than the regular tree view. You can have my SunBurst when you pry it from my cold, dead retinas.

We even make finding and using the API Key easy. BigML’s web interface provides an icon for each resource that lets you get its URL with the api key already encoded, allowing you to access the resource directly from within your application.

However, although the power of your master API Key makes working with BigML’s API easy, it also comes with potential risk. There is no way to share access to your resources in a limited way, and if you do share your master API Key, then you are granting access to every capability in your account. The only method to mitigate this risk previously was the ability to recreate your master key on demand:

In order to address this limitation, our latest release brings the ability to add Alternative API Keys to your account with finer grained controls.  You can define what resources a key can access and what operations (i.e., create, list, retrieve, update or delete) are allowed with it. This is useful in scenarios where you want to grant different roles and privileges to different applications. For example, an application for the IT folks that collects data and creates sources in BigML, another that is accessed by data scientists to create and evaluate models, and a third that is used by the marketing folks to create predictions.

We have implemented some logic behind the scenes to ensure that the permissions you assign are sound. For example, if you want a key to be able to create models, it must also be able to read datasets and models; similarly, if you want your API key to be able to create evaluations it must be able to read datasets, models, and also evaluations.

If you give Alternative API Keys a try please let us know what you think, especially if there is anything we could improve to make it more useful. We appreciate your feedback and are available to help!

Early this year, we showed how BigML ensembles outperform their solo counterparts and even beat other machine learning services. However, up until now creating ensembles with BigML has only been available via our API. We are excited to announce that ensembles are now available via our web interface and that they have also become first-class citizens in our API.

Creating Ensembles

You can create an ensemble just as you would create a model, with the addition of three optional parameters:

1. Whether your want fields to be selected randomly at each split (i.e., decision forest) or only bagging to be used. 
2. The number of models.
3. The task level parallelism.

Decision Forest

A Decision forest or random decision forest is created by selecting a random set of the input fields at each split node while an individual model in the ensemble is being built instead of considering all the input fields. This is the strategy that BigML uses by default. If you just want to use bagging you should deselect this option.

Bagging

Bagging, also known as bootstrap aggregating, is one of the simplest ensemble-based strategies but often outperforms strategies that are more complex. This method uses a different random subset of the original dataset for each model in the ensemble. By default, BigML uses a sampling rate of 100% with replacement for each model, meaning that individual instances can be selected more than once from the dataset. You can select different sampling rates using the sampling configuration panels.

Number of Models

The default is ten, but depending on your data and other modeling parameters you might want to use a bigger number. Generally, increasing the number of models in an ensemble lowers the effect of noise and model variability, and has no downside except the additional cost to you, the user. The cases where more models are likely to be beneficial are when the data is not terribly large (in the thousands of instances or less), when the data is very noisy, and for random decision forests, when there are many correlated features that are all at least somewhat useful.

Keep in mind that each additional model tends to deliver decreasing marginal improvement, so if the difference between nine and ten models is very small, it is very unlikely that an eleventh model will make a big difference.

Task Level Parallelism

The task level parallelism is the level of parallelism that BigML will use to perform a task that is decomposable into embarrassingly parallel tasks like building the models of a random decision forest. We offer five different levels. In the lowest, sub-tasks will be performed sequentially and in the highest level up to 16 sub-tasks will be performed in parallel. The higher the level, the faster your ensemble will be finished. However, the more credits it will cost you.

1-click Ensemble

You can also create an ensemble in just one click.  By default, a 1-click ensemble will create a random decision forest of 10 models using 100% of the original dataset but sampling it with replacement.

Predicting with Ensembles

Once your ensemble is finished, creating a prediction is the same as creating a prediction with a single model, with one additional step; the predictions from the individual models of the ensemble must be combined into a final prediction. The default method for combining the predictions is pluarity vote for a classification ensemble and a simple average for a regression ensemble.

BigML offers three different methods for combining the predictions of an ensemble :

1. Plurality - weighs each model’s prediction as one vote for classification ensembles. For regression ensembles, the predictions are averaged.
2. Confidence Weighted - uses each prediction’s confidence as a voting weight for classification ensembles. For regression ensembles, computes a weighted average using the associated error as the weight.
3. Probability Weighted - uses the probability of the class in the distribution of classes in the leaf node of each prediction as a voting weight for classification ensembles. For regression ensembles, this method is equivalent to the plurality method above.

Predictions take longer in ensembles than in single models, but you can also download ensembles using our download actionable ensemble button to perform low latency predictions directly in your applications. So far they are only available in Python but we’ll bring them to more programming languages soon. We also plan to bring high-performance predictions in an upcoming release, so stay tuned.

Evaluating Ensembles

You can also evaluate an ensemble in the same way as a single model.

The level of accuracy achieved by ensembles of decision trees on previously unseen data very often outperforms most other techniques even if they are more sophisticated or complex.  Not surprisingly, then, it is common to find random decision forests as one of the top performers in Kaggle’s competitions.  Finally, ensembles of decision trees can be applied to perform a multitude of tasks such as classification, regression, manifold learning, density estimation, and semi-supervised classification in thousands of real-world domains. If you’re interested in a great monograph about random decision forests, we recommend this book.

We hope that you give BigML ensembles a try and let us know about your experience and results. Moreover, in this new release there are a number of small goodies like 1-click training|test split, alternative API keys to access to your BigML resources with different privileges, comparing evaluations, and many other things under the hood to make everything come together. We’ll explain it all in future blog posts!