www.RiskNoodle.com

Data Mining Software: Easy-to-Use?

Data Mining Software: Easy-to-Use??  Actually, software available today is designed to be used by the non-experts, i.e., those who do not have an underlying expertise of statistics or mathematics, but DO have both an underlying subject knowledge and basic understanding of data mining concepts.

Data Mining Labs conducted an Evaluation of High-end Data Mining Tools for Fraud Detection looking at the following software vendors: Clementine(ISL), Darwin(TMC), Enterprise Miner(SAS), Intelligent Miner for Data (IBM), Pattern Recognition Workbench (Unica).  The graphic identifies the overall rating for each software application.

The conclusions are as follows:

• IBM's Intelligent Miner for Data has the advantage of being the current market leader, with a strong vendor offering well-regarded consulting support.
• ISL's Clementine excels in support provided and in ease of use (given Unix familiarity) and might allow the most modeling iterations in a tight deadline.
• SAS's Enterprise Miner would especially enhance a statistical environment where users are familiar with SAS and could exploit its macros.
• Thinking Machine's Darwin is best when network bandwidth is at a premium (say, on very large databases).
• And Unica's Pattern Recognition Workbench is a strong choice when it's not obvious what algorithm will be most appropriate, or when analysts are more familiar with spreadsheets than Unix.

Walmart keeps 460 terabytes of data

460 terabytes of data from every single purchase made at the retail chain.  The data does not focus on the who the customer is, but what "basket" of goods that the customer purchased.  This helps Wal-mart in their approach to aisle layouts and placement of items.  A basket can detail your trip around the store and you don't even know you were followed.  This data is gather by state, region, etc.  Its goal...to manage and tighten its grip on the supply and demand chain... sometimes referred to by Wal-mart suppliers as supply and command.

Dick and Jane…See Spot Run. Run Spot Run !

The simple days of grammar school, now long gone, have given way to a myriad of reforms.  The most recent is No Child Left Behind. 

The legislation is designed to change the culture of America's schools by closing the achievement gap, offering more flexibility, giving parents more options, and teaching students based on what works.

Under the act's accountability provisions, states must describe how they will close the achievement gap and make sure all students, including those who are disadvantaged, achieve academic proficiency. They must produce annual state and school district report cards that inform parents and communities about state and school progress.

States are relying upon data mining techniques to identify districts, school, and classroom performance improvement areas. In Oklahoma, a data warehouse, called Class Server, supports district and state reporting needs and provides analysis capabilities used to drive recommendations for improving student achievement based on numerous data points collected from student systems.

With data warehouse integration, Class Server information is readily correlated with actual measures of student achievement. Oklahoma schools can track the long term effectiveness of changes to curriculum, resources and teaching methodologies for individuals or any subgroup.  Analysis and reporting can be done by administrators using familiar tools and interfaces within the existing Microsoft Office environment. Information for students and parents at home or for the community is easily and securely shared through the Web.

Ebay takes the Pulse of it data

Ebay, the world's largest auction house and 12th largest US retailer, annouced two new services to help its customers track and analyze items that sell on its site.  First, is PULSE, a daily barometer of hot selling items, including vendors and prices.  Can you guess the top selling item???? The I-Pod, surprise!

In addition, a subscription service aimed at sellers was launched by Terapeak, called Ebay Marketplace Services. It provides a more indepth look at sales listing using a metrics based analysis.  It can review millions of transactions to develop and provide up-to-the-minute  ratios, price buckets and length of selling periods.  Think of the analysis used on a stock or portfolio and you will have an idea of the type of data delivered with this service.

Since close to 90% of Ebay sellers are "mom and pop" operations, this information will prove extremely valuable to their success.

Lorem ipsum dolor sit amet

Gibberish, you say.  No! It's Latin.  But, it was actually a Privacy Policy used by a "very large" company.  When you clicked on the "Read the Privacy Policy" button, this Latin (or rather Pig-Latin) placeholder was the only information that was provided.  Seems as though Privacy is the last piece of a web design.

However, there are now Best Practices developed for Privacy Statements. By implementing these practices, companies can gain the confidence of the User, protect the privacy of the data , yet allow for effective data collection to be used for further targeting in underwriting, claims management or marketing.

Here are the Guiding Principles developed by P3P:

"P3P has been designed to promote privacy and trust on the Web by enabling service providers to disclose their information practices, and enabling individuals to make informed decisions about the collection and use of their personal information. P3P user agents work on behalf of individuals to reach agreements with service providers about the collection and use of personal information. Trust is built upon the mutual understanding that each party will respect the agreement reached."

Remember, Garbage In - Garbage Out.  The Critical Success Factors for Predictive Modeling begin with solid, accurate data.  Start with Trust!

RED SOX ARE WORLD CHAMPIONS!!!

There is joy in Mudville.
Finally, after sitting through 67, 75, 86 and all the other near misses, the Red Sox have won...in four straight.
Now, can they repeat?!? Can data mining be applied...how about predictive modeling in determining hitters for certain situations, or pitchers to face certain batters, but to include various pieces of data, such as weather, temperature, or tracking players behaviors in certain stressful times.

Yes, even baseball has taken its turn at bat in the field of statistics and modeling. Today, they use sabermaticians to calculate the value of a player as measured in wins. Part of that calculation for hitters relies on a calculation of effectiveness in basestealing. Using data from 1962 to 1977, it can be calculated that with a runner on first and no outs, the expected number of runs in an inning is 0.783. That number increases to just over one (1.068) if the runner can steal and end up on second with no outs. With bases loaded and no outs, the expected runs would be 2.254, with one out, 1.546 and with two outs, 0.798.

It should be noted that the Red Sox hired a Sabermatician at the start of the 2003 season, but really, they only needed a LEADER. Thank you Curt Shilling!

"Anything that exists exists in a certain quantity and can be measured"

In his famous statement of 1904, Edward L. Thorndike makes the case for measurement in educational instruction and research. However, his truism, is especially relevant today when data from all sources is being used in predictive modeling equations. Perhaps, one of the major challenges is how to adequately quantify human behaviors. For it is precisely this information that will help predict life expectancies, fraud tendencies, not to mention buying/selling tendencies.

Don't have enough data....try text mining

There is no question that the more data at your disposal coupled with the high quality value of that data allows for an accurate forecast or prediction. However, when dealing with issues that aren't neccessarily reflected in traditional patterns, it may be useful to deal with term extraction or "text mining".

By employing OCR scanning and assigning a weighted value for certain words, the model can score as well as identify recurring words and compute a value to alert claims professionals to patterns or indicators of fradulent activity.

GIGO...

Garbage in...Garbage out....

In today's world of information, it may be nearly impossible to know which set of data can lead you through to anticipate trends in your industry. Clearly, identifying the correct data can help cross sell as well as understand and enhance the sales and operating margins. It can also help you quickly change or adopt new strategies based upon the correlation of data.

First, you need to understand the very basics about data mining and it resultant statistical processes. With software on the market today, you need not become a statistician, but true value comes into play by understanding the application and effects of data mining. It has now become a reality for small to large businesses to bring the power of data to fruition.