Open Search with Skrodon

Speaker: Mark Overmeer

Abstract

Search for the global internet is a very impactful and especially very lucrative business. Billions are spent to reach top ranking results with good content or ads. Although many companies, and researchers want to contribute to the search on internet, there is no chance to participate in any sufficiently scaled application.

The presented projects try to build fundamental building bricks for cooperation towards an open search infrastructure. Together, we can improve our own products and slowly build towards a generic public search engine on full scale.

We will present three components:

• Skrodon is our working title for an association of groups which offer Open Services and Data, strictly under EU privacy and fair competition law;
• Open Console tries to coordinate the collection of information about websites and domain information, an Open competitor for Google's Search Console; and
• the Crawl Pipeline, which can scan 10TB of crawled web-pages per day, to extract the information your application needs.

These projects are sponsored by the NLnet Foundation, and developed in cooperation with the Open Search Foundation.

Biography

Mark wrote his first program in 1978, and had one of the first email addresses and one of the first websites in the Netherlands. In the past 20 years, he worked as free-lance programmer, devoting half of his time to (usually unpaid) open source development.

He has contributed many large modules to Perl, (co-)organizes about 10 events per year, and visits many more. Besides, he is "Buro NLUUG".