traims' blog

Copula-based clustering of datasets with multivariate dependence structure: a brief overview and links

Wed, 16 Jan 2013 20:18:00 +0100

I have just attended a talk on copula-based clustering algorithms, where work of Francesca Marta Lilja Di Lascio and Simone Giannerini was presented.

Copula-based clustering is a kind of model-based clustering where each cluster is modeled as a set of realizations of one random variable. To model k clusters, k-dimensional copulas are used. Different kinds of copulas can be used for modelling the data; it’s up to the researcher to decide which ones to choose.

The method is particularly useful for datasets with a high dependence between different clusters. Its performance degrades when dependence is lower.

The R package CoClust, which implements a new clustering method based on copulas, is available on CRAN: http://cran.r-project.org/web/packages/CoClust/ (R Help)

References:

F. Marta L. Di Lascio, Simone Giannerini. A Copula-Based Algorithm for Discovering Patterns of Dependent Observations. Journal of Classification. April 2012, Volume 29, Issue 1, pp 50-75
Di Lascio, F. M. L. (2008). Analyzing the dependence structure of microarray data: A copula–based approach. Ph.D. Dissertation, University of Bologna, Bologna, Italy. Available at http://amsdottorato.cib.unibo.it/670/

Python for Data Analysis, 18 Oct 2012, London

Fri, 19 Oct 2012 09:13:00 +0200

I have attended “Python for Data Analysis” meeting organised by Data Science London. There were two main talks — by Didrik Pinte from Enthought and by Wes McKinney, creator of pandas.

NumPy, the Python foundation for number crunching

by Didrik Pinte @dpinte — Python contributor to QuantLib (a library for quant finance), and MD of Enthought, developer of EPD-the scientific computing Python platform.

(c) by Data Science London ‏@ds_ldn

About a half of the audience has already used NumPy, though I think only a couple of people has gone deep with C integration and memory optimizations. So it was a mix of an introductory talk with showing Cython code and profiling tools.

Interestingly, when someone decided to port NumPy to .NET, it didn’t work efficiently because of unpredictable garbage collection in .NET.

Didrik has also shown how a memory monitor from Pikos works.

Python for Data Analysis

by Wes McKinney @wesmckinn — former quant, author of pandas (the powerful Python library for data analysis), author of the book: “Python for Data Analysis”

(c) by Data Science London ‏@ds_ldn

Most of the talk was done in the ipython notebook. Using a MovieLens dataset as an example, Wes has shown different pandas functions: data slicing, merge, map etc. The library is also good for data munging/cleaning/preparation.

He told they are doing further improvements of the library because of use cases when people try to open a 5 GB Kaggle dataset and the system uses 20 GB of memory.

Rmagic library: running R code in Python. Useful e.g. for ggplot2 library, which has no matches in the Python world.

“Python for Data Analysis” book is an introduction to pandas with working code examples, a better learning material than plain documentation. Print copies are not available yet; books will probably appear for Strata New York. (I have just checked my O’Reilly account, my copy is not listed as an “early release” anymore).

The first speaker, Didrik, added two reasons to use pandas:

its excellent documentation;
vbench, performance benchmarking for pandas.

Random notes:

Carlos, the event host, offered three O’Reilly books as quiz prizes. Didrik came up with a question about ndarray strides. Wes asked who has created the J language, which inspired NumPy.
Didrik was using Canopy on his Mac. Wes was using ipython notebook.
Meetup page of the event: http://www.meetup.com/Data-Science-London/events/85448442/

OKFN and the European Journalism Centre

Tue, 16 Oct 2012 23:53:18 +0200

Open Interests Europe Hackathon will take place in London, November 24 and 25.

Open Interests Europe brings together developers, designers, activists, journalists and other geeks for two days of learning, fun, intense hacking and app building.

How EU money is spent is an issue that concerns everyone who pays taxes to the EU. As the influence of Brussels lobbyists grows, it is increasingly important to draw the connections between lobbying, policy-making and funding. Journalists and activists need browsable databases, tools and platforms to investigate lobbyists’ influence and where the money goes in the EU. Join us and help build these tools!

The Hackathon challenges include Lobbying Transparency and Fish Subsidies.

Organised by the Open Knowledge Foundation and the European Journalism Centre, sponsored by Knight-Mozilla OpenNews.

For more details, please see the event website:
http://okfnlabs.org/events/hackdays/lobbying.html

and register at http://openinterests.eventbrite.com

Yuri Suzuki: London Underground Circuit Map images © hitomi...

Fri, 14 Sep 2012 08:07:00 +0200

Yuri Suzuki: London Underground Circuit Map

images © hitomi kai yoda

Japanese designer Yuri Suzuki has sent DesignBoom images of his ‘London underground circuit maps’ project, developed as part of the Designers in Residence program at the London Design Museum, on show until January 13th, 2013.

Palo Alto looks to use open data to embrace ‘city as a platform’ -- O'Reilly Radar

Sun, 05 Aug 2012 17:29:00 +0200

“The city of Palo Alto in California joined over a dozen cities around the United States and globe when it launched its own open data platform.

The city initially published open datasets that include the 2010 census data, pavement condition, city tree locations, park locations, bicycle paths and hiking trails, creek water level, rainfall and utility data. Open data about Palo Alto budgets, campaign finance, government salaries, regulations, licensing, or performance — which would all offer more insight into traditional metrics for government accountability — were not part of this first release.

The platform includes an application programming interface (API) which enables direct access through a RESTful interface to open government data published in a JSON format. Datasets can also be embedded like YouTube videos.”

— Palo Alto looks to use open data to embrace ‘city as a platform’, O’Reilly Radar

Data samples

CSV kit: a set of command line tools for handling CSV files

Sun, 05 Aug 2012 16:25:00 +0200

CSV kit: a set of command line tools for handling CSV files:

csvkit is to tabular data what the standard Unix text processing suite (grep, sed, cut, sort) is to text.
csvkit is designed to augment or supercede much of Python’s csv module.
The csvkit tutorial walks through processing and analyzing a real dataset from data.gov.

Repository: https://github.com/onyxfish/csvkit
Documentation: http://csvkit.rtfd.org/

via @markw

London: A Twisted Underground Map | Minefield Junction

Sun, 05 Aug 2012 10:16:00 +0200

London: A Twisted Underground Map | Minefield Junction:

It’s a rather strange version of Harry Beck’s London Underground map, in which I took the approach of “reflecting the topology, ignoring the geography” to the limit.

EuroPython started today.

Mon, 02 Jul 2012 18:42:00 +0200

T-shirts and MongoDB mug

Coffee break for 700+ people

Guides and leaflets

Waiting for the opening

EuroPython started today.

UCR Insect Classification Contest

Sun, 01 Jul 2012 13:55:36 +0200

www.cs.ucr.edu/~eamonn/CE/contest.htm

“Dr. Keoghs lab is hosting a contest to create a similarity measure for insect flight sounds (there is anecdotal evidence that music similarity algorithms may work).

If we could correctly identify insects using cheap sensors, we could:
1) Plan malaria interventions more effectively, perhaps saving some of the million human lives lost each year.
2) Plan insect pest crop interventions more effectively, thus growing more food with less pollution, less energy, less environmental damage etc.”

Phase I: July to November 16th 2012
• The task is to produce the best distance (similarity) measure for insect flight sounds.
• The contest will be scored by 1-nearest neighbor classification.

Simeon Lobo: Useful Hadoop and MapReduce Algorithms

Sat, 23 Jun 2012 15:03:41 +0200

Simeon Lobo: Useful Hadoop and MapReduce Algorithms:

simeonlobo:

I’ve chosen to reproduce the below list from Amund Tveit’s article so I can maintain a backed-up personal reference.

I also intend to update this collection of Hadoop MapReduce algorithms based on my growing experience with the platform.

Artificial Intelligence / Machine Learning / Data…

Add github gists to your tumblr

Sat, 23 Jun 2012 14:49:44 +0200

mutatedmonkeygenes:

https://gist.github.com/1395926

Papers on massive data mining

Sat, 23 Jun 2012 11:47:58 +0200

The page of the Summer School on Massive Data Mining (August 8-10, 2012, Denmark) lists several papers as recommended reading. Namely:

Mining patterns from large datasets:

Matrix and graph algorithms:

Recommended reading for two more talks (mining large graphs, and clustering/metaclustering) is to be announced.

Hack/Reduce: another kind of business incubator

Tue, 19 Jun 2012 18:19:00 +0200

Thanks to a Twitter recommendation, I stumbled upon Hack/Reduce, a non-profit initiative from Massachusetts.

Hack/reduce is a community hacker space in Boston. We provide a space where people working with Big Data can come to share infrastructure, resources and knowledge.

Seems quite interesting. From a broader point of view, this is yet another approach for creating a “business incubator”.

Hack/Reduce will develop the necessary talent to create companies and jobs to shape the future in the Big Data driven economy.
It’s about hands-on learning: tools, techniques, best practices, etc.

It’s also about connecting people that don’t normally connect: super geeks, business, artists; people that are passionate about content.

— “The most important job in Boston”: A Q&A with Chris Lynch on Hack/Reduce

NodeXL visualizations: conference Twitter posts and GitHub data

Tue, 19 Jun 2012 15:49:00 +0200

NodeXL is a tool for working with network graphs, developed as a free open-source template for Microsoft Excel. Examples of visualizations created by the community are stored in the NodeXL Graph Gallery.

I found these two visualizations from the library particularly interesting:

Collective Intelligence Conference

A network graph based on Twitter data created by Ben Shneiderman. Apparently, it shows links between Twitter accounts of people who posted about the Collective Intelligence conference. Clustered by the Clauset-Newman-Moore algorithm.

[link to the gallery entry]

Related: Anatomy of a Conference Twitter Hashtag: #AALL2010 (PDF + data)
#jisccdd Hashtag Analysis in Google Docs

GitHub: organizations and programming languages

The next visualization created by Eduarda Mendes Rodrigues shows links between GitHub organizations and the programming languages. “Each link between an organisation and a programming language indicates that the organization has at least 250 projects written in that language.”

[link to the gallery entry]

Results are sometimes unexpected, at least in the terms of visialization. CoffeeScript turned out to be much closer to PHP than to Ruby. Prolog appears near Java (probably because of Java libraries for logic programming?). Assembly landed somewhere between C and Ruby. Anyway, “at least 250 projects” condition may appear a little bit artificial: only large organizations are considered.

1.USA.gov: public data on shortened links to .gov and .mil websites

Tue, 19 Jun 2012 15:40:00 +0200

I am currently reading an early release of “Python for Data Analysis” written by the creator of pandas. It is an unfinished version, so some of the chapters are missing, and there are a lot of TODOs around. Full version is planned for release in October 2012. Nevertheless, it is quite interesting to read even now.

From this book, I found out about 1.USA.gov, open data about URLs in .gov and .mil domains shortened by bit.ly. They provide anonymized data created anytime anyone clicks a shortened link pointing to a .gov or .mil address.

You can get this data from a live feed or download an hourly snapshot. A JSON entry looks like that:

{ “a”: “Mozilla\/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) AppleWebKit\/534.46 (KHTML, like Gecko) Mobile\/9A405 Twitter for iPhone”, “c”: “CA”, “nk”: 0, “tz”: “America\/Winnipeg”, “gr”: “MB”, “g”: “JKZUHq”, “h”: “J8ZPYk”, “l”: “nasatwitter”, “al”: “en-us”, “hh”: “go.nasa.gov”, “r”: “https:\/\/twitter.com\/ nasa\/status\/204658186135932930”, “u”: “http:\/\/www.nasa.gov\/ mission_pages\/hinode\/eclipse_120520.html”, “t”: 1337634445, “hc”: 1337629186, “cy”: “Winnipeg”, “ll”: [ 49.883301, -97.166702 ] }

Further information can be found in USA.gov Developer Resources.

The last year they even conducted a Hack Day to encourage people to dig into this data. The most interesting results were NASA Shuttle Launch Visualization and the mere fact that 42% of all clicks link to NASA websites. People from Europe are more likely to visit NASA than other websites. Still, I wonder what drives the collection of all these data.

Crawling the Web to Keep It Open

Thu, 24 May 2012 18:12:00 +0200

Speaker: Lisa Green (Common Crawl)Common Crawl builds and mantains an open crawl of the web that anyone can use. A lot of possible applications in research and business.

“Common Crawl aims to change the big data game with our repository of over 40 terabytes of high-quality web crawl information into the Amazon cloud, the net total of 5 billion crawled pages”.

Datasets are updated with a high frequency. Additionally: tutorials, YouTube videos, GitHub presence.

The foundation is particularly interested in working with academia. Web Data Commons is a collaborative project with two German universities.

Liberating Data @ Wikipedia

Thu, 24 May 2012 18:12:00 +0200

Speaker: Diederik van Liere (Wikimedia Foundation) Ecosystem built around Wikipedia is huge. “Open, share, attribute” is in Wikimedia’s DNA. But they can make a better job with data: releasing more datasets and developing tools and APIs so that people could use it. Wikimedia is a small foundation, they need to crowdsource. Data should be more machine-readable. “All capitals of all countries in the world”: this question can be answered by DBPedia, not Wikipedia. Tools:

Wikihadoop is a tool for processing compressed XML dumps of the Wikimedia projects. Output is the difference between two revisions: what added? what removed?
RevDiffSearch is a Lucene-based full-text search engine. Can answer questions such as “What authors are adding geo-templates? Who is adding citations?”
KRAKEN is the Wikimedia analytics system. Development in progress. Hadoop, Cassandra, Hive, Zookeeper.
Wikimedia Report Card

Datasets:

Wikimedia Dumps: pageviews, surveys, XML dumps.
http://www.datahub.io/dataset?q=wikimedia

Data Philanthropy: Case Studies

Thu, 24 May 2012 18:10:00 +0200

Hosted by: Jake Porway (DataKind)DataKind organizes weekend projects, hackatons, nightime projects to support NGOs. Why it works:

For NGOs: better decision support. Small teams cannot afford to hire a data scientist.
For software engineers: helping people to make world a better place is more exciting than coding typical hackaton apps (such as parking apps).

Three case studies with three different organizations:

Community Knowledge Worker initiative Speaker: Emily Tucker (Grameen Foundation) Farmers in poor rural areas have no basic information about market prices or weather forecasts. Grameen Foundation recruits people directly from the community, and they are trained to use a smartphone. Then “knowledge workers” literally go from farmer to farmer to share information. Grameen collects a lot of data: GPS location, demographic information on participants, size of their farms. This data can help to estimate probabilities (what is more likely for families with 11 children? Are bicycles too expensive?) As they cannot afford a data scientist, they work with DataKind to identify very specific questions to support concrete decisions about how to improve things. Even a weekend project can have a real impact. Related: Slides: Using Data to Drive Decision Making, March 15, 2012. Emily Tucker, Grameen Foundation
Helping children to reach their full potential Speaker: HyeSook Chung HyeSook Chung (DC Action for Children) “In the areas of concentrated poverty, traditional way of thinking will not take us too far. We need untraditional approach”. Their choice is data-driven approach. Depending on where a child lives, we can predict how it will succeed in life. This can push the city to think about allocation of resources. DataKind and their volunteers created a prototype visualization of conditions that exist in different neighbourhoods in Washington, D.C. It shows visually on the map childcare facilities, libraries and other relevant information that helps to answer the question: is this neighbourhood friendly for raising children?
Identifying high-performing non-profits Speaker: Braz Brandt (GuideStar USA) GuideStar gathers and publishes information on non-profits. There are a lot of data on non-profits, and a goal is to make it available online. It is possible to build predictive models for non-profits based on the data collected on them. The ultimate goal is to identify high performing non profits. “Are they able to use the money effectively? Are they likely to fail?” If we support the best organizations, they are more likely to make changes happen in the world.

Open Knowledge Foundation: Open Tools for Open Data

Thu, 24 May 2012 15:55:00 +0200

Speaker: Rufus Pollock (Open Knowledge Foundation) The Open Knowledge Foundation is a not-for-profit founded in 2004 and particularly active in Europe. They focus on open data. Problem-oriented approach: using digital information more effectively to improve governance, research and economy; to build more sustainable cities. Challenges: tools and datasets are diffucult to find and use. Datasets don’t benefit from a larger community engagement. They developed their own tools to address these challenges. Tools: Comprehensive Knowledge Archive Network (CKAN) Software:

CKAN is a content management system that makes it easy to publish, find and use data.
Free open source software; you can download it for free and host it on your servers.
Source code is available in Python package index. Documentation is here.
Extensible and flexible; add-ons for stats, analytics, data APIs etc.
Dozens of CKAN deployments around the world: data.gov.uk, DataHub.io …

DataHub.io is community data hubpowered by CKAN. Like GitHub is for code, DataHub is for data.

publish a dataset: upload a file (or use CKAN API…)
search for datasets
visualize: charts, geomaps… (search by region)
data API for external data access (documentation available at docs.ckan.org)

Recline.jsis a stand-alone JavaScript library for data vizualization. Built with Backbone.js.

Data Explorer: “combines a data grid, Google Refine-style data transforms and visualizations.”
A library of data components: “data grid, graphing, and data connectors.”

Other projects:

OpenSpending.org: financial information from different countries and sources.
Where Does My Money Go (UK)
School of Data: community-oriented project; kick-off meeting May 24 (in Berlin and online).
The School will provide online training for data ‘wrangling’ skills – the ability to find, retrieve, clean, manipulate, analyze, and represent different types of data. Everyone is welcome, and we will try to have a range of hands-on activities to suit everyone’s interests.
Frequently Asked Questions about School of Data

Narrative Science: Transforming data into stories

Thu, 24 May 2012 15:52:00 +0200

Strata Online Conference, Speaker: Kristian Hammond (Narrative Science)

Narrative Science transforms data into narrative content in plain English at an enormous scale. Here is a sketch of how it works. Right now Narrative Science works with media companies and data companies; looking to move on government/public policy data.

Tools: Python, MongoDB, MapReduce.

So far, their toolset is for internal, not external use.

Media coverage of their business: Wired, O’Reilly Radar