Scaling up globally:30 years of FOSS4G development
Keynote
Markus Neteler
Fondazione Edmund MachResearch and Innovation CentreDepartment of Biodiversity and Molecular EcologyGIS and Remote Sensing Unit
FOSS4G Central and Eastern Europe 2013
16th - 20th June, National Library of RomaniaBucharest, Romania
PostGISomics
Good afternoon and welcome to my talk with the bold title
Scaling up globally: 30 years of FOSS4G development
I am very happy to be here, and would like to thank the organizers for the kind invitation.
In my presentation I'll briefly review 3 decadesof Open Source GIS development, compressed to 30 minutes! Hence, omissions are unavoidable. BTW: The term FOSS4G was coined only in 2004 but I use it in the title as an hommage to the community and its ideals.
So, let's now look back 30 years back to the 80th...
Img. courtesy: J Westervelt 2006:
Early GRASS Community Views on FOSS
THE 80's the beginning
Well... what's this? Do you recognise it?
THE 80's ...
1980 LAGRID J. Westervelt master's thesis:
GIS software, developed on a mainframe computer,
then ported to Cromemco Z-80. FORTRAN
1982 FHIS (Fort Hood Information System)
Vax 11/780 minicomputer, UNIX, C language. Programmer: L. Van
Warren
1983 "GIS Version 1 Reference Manual"
by J. Westervelt and M. O'Shea, 29 July 1983
Included GIS programs:
1983 PROJ4 library development started by Gerald I. Evenden
arctogrip area_stats cell_stats cellmod (grid editor) coin
(r.coin) combine (boolean combination) distance (r.buffer) dotmap
(graphics on a
dot-matrix printer) erase (d.erase)
griptocell layer_info (r.info) list (g.list) over (d.rast, but for b/w monitor) reclass (r.reclass) sho_over (display images created by over) table (stats associated with over) whats_here (r.info with a mask) window (g.region)
Vax 11/780minicomputer
Hence we start in 1980: A student named Jim Westervelt was completing his Master thesis. He wrote the LAGRID software which became the core of the later GRASS GIS software.In the early 1980s the U.S. Army Corps of Engineers' Construction Engineering Research Laboratory (USA/CERL) in Champaign, Illinois, began to explore the possibilities of using GIS to conduct environmental research, assessments, monitoring and management of lands under the stewardship of the U.S. Department of Defense.
Bill Goran of CERL conducted a survey and discovered that no existing GIS satisfied their needs. Hence CERL hired several programmers, and began to develop a hybrid raster-vector GIS for the VAX UNIX minicomputers (photo). Lynn Van Warren was the founding software architect of GRASS.
Very important: PROJ (today PROJ.4) the Cartographic Projections Library essential for many projects, was started in 1983 by Gerald Evenden
Boolean combination - gridcell and polygon Weighted overlay - gridcell and polygon (reworked later into r.mapcalc) Distance-from (now r.buffer) Isoline generation (grid to poly - now r.contour) Coincident tabulation (now r.coin) Mathematical combination (reworked later into r.mapcalc)THE 80's ...
August, 1983 U. S. Army Corps of Engineers'
Construction Engineering Research Laboratory
(USA/CERL) in Champaign, Illinois
Purchase of first Sun-150 computers (2) specs:$8.900 16 bit,
10Mhz, 256Kb memory, 10Mbps Ethernet,
17" b/w monitor (100*800)
$3.400 1Mbyte extra memory
$2.000 3/4M Fast Sun Memory
$1.590 Barko color monitor
$6.540 80Mbyte hard disk
$1.900 Disk controller board
$5.500 Dot Matrix printer
$1.500 Vanilla UNIX software -or-
$2.000 4.2BSD plus library of graphics software
1983/1984 New analysis capabilities added in GRASS GIS:
Source: http://lists.osgeo.org/pipermail/grass-psc/2012-December/000985.html
33k USD
CERL then upgraded their machines to the latest technology, see
what you could get for 33,000 USD.... e.g. including a 80MB disks
for 6500 USD.
Around 1984 new analytical functionality was available, some
command names at least the GRASS users in this room will be
familiar with :-)
Please note that it took still another almost 10 years to have the World Wide Web being invented...
THE 80's ...
1984 GRASS running on SUN-1 and Masscomp
1985 GRASS 1.0, GRASSnet established
(pre-mailing list)
15 March 1985: first commercial Internet domain name registered
1985 Richard Stallman publishes the GNU Manifesto
1987 GRASS 2.0; first issue of GRASSClippings Newsletter GRASS video narrated by William Shatner (Captain Kirk of Star Trek)
1988 GRASS 3.0; Army R&D Achievement Award (Webster, Goran, Shapiro, Westervelt)
In the corner photo you see what data exchange meant in those days. Landsat 5 was launched in 1984, GRASS 1.0 was released a year later.
Of huge importance for the Free Software development was the publication of the GNU manifesto by Richard Stallman (the four freedoms were defined).
In 1987 a GRASS video was produced with William Shatner, known as Captain Kirk of Star Trek, being the speaker.
And 5.25 inch floppies were becoming fashion in these days for tiny data sets!
THE 80's ...
1989 GRASS 3.1
First release available on Internet
(uxc.cso.uiuc.edu)
But how was software developed? Locally!
While Revision Control System (RCS) was available and
sometimes used, yet no server based system like CVS(CVS 1.0 in 1990
today: SVN, git, )
TCP/IP goes global (19892010)
However:
By 1992, still less than 15,000 .com domains registered...
See also: http://grass.osgeo.org/home/history/
http://wiki.osgeo.org/wiki/Open_Source_GIS_History
http://grass.osgeo.org/uploads/grass/history_docs/westervelt2004_GRASS_roots.pdf
And then finally an initial internet release of Open Source GIS software! Still there was almost nobody online in those days, at least not able to download 100 MB of source code...
The big spread of internet happened due to the acceptance of TCP/IP as protocol.Information exchange happened via FTP.
Today all geeks are sharing code via SVN or git or other online code sharing repositories. But back in the 80th there was no online system like this. Either manual management or at most RCS which is file based.
THE 90's visualization
and analysis
Source: Helena Mitasovahttp://skagit.meas.ncsu.edu/~helena/gmslab/gsoils/vizrep2.html
So, the first decade of Open Source GIS concluded with the online publication of software (yet with a tiny user group being also online).
Consider to think back when you went online the first time!
Anyway, let us hop into the next decade:
the 90th!
A lot happened here, new hardware, new software, and way more data!
And especially the advent of the internet society was a social revolution.
THE 90's ...
1990 GRASS 4.0 Letter-dot format adopted for commands (e.g. d.rast and g.region)
1991 First mailing lists
In 1990 GRASS 4.0 came out, a key release which was the foundation for the GRASS' architecture.
On the social side, two mailing lists were started, the grass-user and the grass-dev lists. Those are probably the longest, active mailing lists of the internet with more than 2 decades of archive!
Still no WWW, but information exchange happened FTP and Gopher.
The photo shows core developers:- Michael Shapiro- Jim Westervelt- Bill Goran (the coordinator)
THE 90's ...
1992 New graphics technology became available: openGL
Software distribution:Moon FTP server of CERL
(note: still 2 years to wait for WWW!)
SG3D
SGI Indigo
With the availability of new graphics CPUs a completely new way of rendering became possible. OpenGL was designed with hardware acceleration.
Like this, fly-thoughs as seen in the left figure as well as 3D and even 4D animations came to life (here, water contamination modeled as voxels which are changing in time).
I remember from working as a student at the Hannover CeBIT computer fair, to enjoy the demos of the SGI machines while carrying around newspapers across the fair area.
THE 90's ...
ORGANIZATIONS
1992GRASS Inter-Agency Coordinating Committee Open GRASS Foundation (OGF) OpenGIS Consortium (OGC)
1994 Tim Berners-Lee decided to constitute the
World Wide Web Consortium (W3C)
1994 Open Geospatial Consortium
(OGC) founded
Open GIS Open Geospatial
Consortium (OGC) Consortium (OGC) GRASS Interagency
Steering Commitee1990
1992
2006
Open GRASS
Foundation (OGF)1994
2004
OSGeo
At the level of organizations many changes occurred: the GIS world became more professional and organized.
In 1992 the GRASS Interagency Coordinating Commitee was founded, it was turned into the Open GRASS Foundation which then became the OpenGIS Consortium.
The timeline shows the evolution.
While Tim Berners-Lee constituted the World Wide Web Consortium, the today's OGC was established.
THE 90's ...
http://grass.osgeo.org/home/history/documents/
The rationale behind the establishing of the OGC you find described in an article written by Kenn Gardels. It got published in the GRASSCLIPPINGS issue of Fall 1993.
Here the fact that interoperability is a core issue is pointed out.
BTW: If you are interested, the scan is online, all links in my presentation which I'll upload later, are pointing to the respective documents.
Source:http://news.bbc.co.uk/2/hi/technology/8552410.stm
THE 90's Internet growth as a crucial
precursor to FOSS4G development
This map shows the internet accessibility in the year 1998 people being online in percent.
As you see, in those days internet was yet restricted to a few countries in terms of accessibility.
And, obviously:without internet no distributed source code development nor easy geodata exchange.
Do you remember:
30 slackware disks downloaded,but unfortunately in FTP ASCII
mode (7 bit, not 8)??
THE 90's ...
First Web Applications come up, finally!
1993 Xerox PARC Map Viewer
1994 GNU/Linux 1.0 released
1995 First GRASS 4.1.5 port to Linux
(Andreas Holz, Greifswald, Germany)
1995 UMN MapServer project started
http://wiki.osgeo.org/wiki/Open_Source_GIS_History
1995 Well, and also the first spam email in the GRASS mailing list
In the year 1993 the first Web application came up, the Xerox
PARC map viewer
[refer to the previous talk of Maria Brovelli]
In 1994 the 1.0 version of Linux was released and shortly after GRASS GIS ported to it. This was still a tough job since the compiler tools where not as mature as nowadays.
The year 1995 was a key year for the WebGIS community with the start of the Mapserver project.
And... wow, a first spam email reached the ML
THE 90's ...
An interactive GIS via WWW, perhaps the first Web Processing service...
1994-1998 Susan M. Huse's GRASSLinks (PhD thesis at Berkeley)
What happened in the WebGIS sector:
Probably less known is the this first Web Processing style service which was published back in 1994.
It was an interactive GIS controllable through the WWW, developed by Susan Huse based on her PhD thesis.
This system was able to ...
THE 90's ...
GRASSLinks
perform true analysis in a user controllable way using CGI scripts.
Of course it is not comparable with anything you can do today but remember the internet accessibility map of 1998 which I have shown earlier.
We are still in the prototype years and draft WPS specifications were published only almost a decade later.
THE 90's ...
Internet, its tools and geeks spreading!
1996 GeoTools project started
1998 deegree (originally JaGo) development started
with an OGC Simple Features implementation
1998 GDAL/OGR development started
1998 First European GRASS GIS server at ILN, Uni Hannover, Germany
1999 GRASS GIS source code moved from manual management to
CVS,
precisely on 29 Dec. 1999 :-)
With the advent of the collaborative internet based development tools, a series of new projects like GeoTools, deegree, GDAL/OGR were started.
In 1998 I started the European GRASS GIS server, completely non-authorized of course (except for the ILN landscape architecture institute at the university)... it turned out to be a good idea to take control.
BTW: Anyone here remembering the year 2000 bug?It stimulated us to move the GRASS code from manual code management via email to something more sophisticated called CVS. This was a server based code repository and checkins became independent of a single person who had to read emails and act upon.
Since 2000...
May the FOSS be with you
Well, we happily survived the Year 2000 bug and reached the next decade...
Now Communities growing together!
This drawing has been done by the older daugther of Venka (Venkatesh Raghavan in Osaka) for the initial 2004 FOSS4G conference in Bangkok which I'll mention shortly
2001 OSSIM initial revision in CVS PostGIS started GeoNetwork opensource started GeoServer started2002 Quantum GIS initial revision in CVS GEOS initial revision in CVS2003 Community MapBuilder started. End of life in 2008 Release of Mapbender under the GNU GPL license gvSIG was started 2004 uDig was started2005 MapGuide Open Source2006 Mapbender gets first bits in CVS OpenLayers Started2007 GeoMoose was open sourced (started 2005)2009 rasdaman was open sourced (started 1995)
Since 2000... growing communities
Thanks to CVS,
SVN and git2006: OSGeo!
http://wiki.osgeo.org/wiki/Open_Source_GIS_History
In just a few years a lot of new software projects were started, I don't have the time to illustrate them in details. You see that all adopted the collaborative tools for development.
From the demand to get coordination among these projects the idea arose to create an umbrella foundation, OSGeo, as illustrated by Jeff McKenna earlier today.
Since 2000...growing communities
Chulalongkorn University,Bangkok 2004:
FOSS4G is born!
Jim Westervelt
Special thanks toVenkatesh Raghavan,Osaca City University
For the term FOSS4G we have to thank Venka once more. He proposed the conference title in 2004 for the conference at Chulalongkorn University.
Also Jim Westervelt came to give us an authentic lecture about the early days at this meeting.
Since 2000...growing communities
QGIS Hackfest Pisa 2010
Mapbender
code sprint 2007
FOSS4G Conferences:Lausanne, Denver,Victoria, Cape Town,
Sydney,
Barcelona, Denver, ...
The next slides will be much more familiar for you... code sprint photos
Here the mapbender team in 2007 and the QGIS hackfest in Pisa in 2010.
Community sprints, even more...
For those not familiar with it:- it is a gathering of likeminded people- no need to be a developer- newcomers are there up to core developers
Instead of writing 10 emails to discuss an issue, we just discuss it directly, rarely long with some beer
And: The outcome of such a week is often impressive.
Next chance for you: on Thursday this week!
CommunicationFlow of bug reporting and solution:
2. Developer
detects bug
60%
20%
20%
(Percentages are estimated)
Other
developers
New
release3. New
feature
SVN code
RepositoryReleasecycle
1. User sends bug reportBugtrackerMailing list
20%
80%
Organization of distributed source code
management: Code habitats
Two main types of developers may be identified:generalist
specialist (the majority)
It appears that many developer assign themselves to
code habitats, i.e. their area of expertise (e.g., in GRASS
GIS
a selection of libraries or topics which they maintain)
These code habitats remain often stable over years
There are also partially abandoned code areas (~ 10% of the
code?)
which are functional but aren't really getting improved
A few garbage collectors (generalists) fix lots of odds 'n ends
Organization of distributed source code
management: Code habitats
http://www.youtube.com/watch?v=suyDqmGXoWk
GRASS GIS 6.4 development visualization from 1999 to 2011 with Gource
Code vetting
Legal aspectsLicense compliance (e.g., GPL)No code copying from
books like Numerical Receipes in CEnsure that 3rd party
contributions are cleanEmployers must agree that work time is
spent
Full transparency and peer review help to minimize the risk.
Apache or OSGeo FoundationIncubation phaseGraduation
http://incubator.apache.org/
http://www.osgeo.org/incubator
Since 2010...
Source: Blog of Arnulf Christl
Emergency support
Haiti, January 2010:
7.0 magnitude earthquake
Available Geodata:
almost absentSee also:
http://hot.openstreetmap.org/projects/haiti-2
Haiti Dec 2009
Open Data: OpenStreetMap.org
After these years of community based software development, the emerge of Wikipedia also crowd mapping came up.
No need to explain OpenStreetMap here but for sure the Haiti earthquake example is to be mentioned.
It is illustrating an outstanding effort of people around the globe gathering virtually together and helping in an emergency situation.
The map shows the geodata situation in January 2010 when a magnitude 7 earthquake occurred almost no geodata available which were urgently needed.
A few days later ..: international crowd mapping
See also: http://hot.openstreetmap.org/projects/haiti-2
Haiti Jan 2010
Open Data: OpenStreetMap.org
This is the situation a few days later. Based on high resolution satellite data which were made available by the big data providers, the OSM community was able to generate in an extremely short time a detailed coverage of digital data.
A similar effort was done after the Great East Japan Earthquake 2011, you will remember it.
New: Crowdfunding of development
The increasing demand for geospatial data in the Web browser and increasing interactivity naturally requires the adoption of new emerging technologies.
An interesting new method of fundraining, instead of direct sponsoring by a few is after crowd knowledge (Wikipedia) and crowd mapping (OSM) now crowd funding where till a deadline a certain funding goal should be achieved.
Example: OpenLayers 3 funding, aiming at more than 300,000 USD and they are almost there!
New cool stuff: plotting GIS data
From
Radar
to
voxels
to
3D plots
There are also new possibilities for plotting maps: from 2D to 3D using rapid prototyping technologies which have been directly connected to Open Source GIS.
Here an example of the polar ice cap of Mars (so, no need to restrict yourself to mother Earth):
The processing chain was: from Radar remote sensing data to voxels to 3D plots
New cool stuff:
massive data processing
Since 2005 GRASS GIS is running natively on 64bit CPUs
GRASS GIS 7 also offers Large File Support on 32bit Windows
Installed on Grids and TOP500 supercomputers (AKKA Ume,
ENEA Frascati, Aurel Bratislava, )
Runs on Linux, AIX, Solaris, freeBSD, netBSD, ...
Various ways of parallelization
Since we like to think big, why to getting our tools on Supercomputers running...
Indeed, we are already there:PROJ4, GDAL and GRASS GIS are available even on TOP500 systems.
The harder part was to get it compiled on Non-Linux Supercomputers :-)
For massive data processing currently job managers are supported. Work is underway for GPU based clusters which require a major source code restructuring.
Massive data processing:
also for you
https://plus.google.com/u/0/communities/111147786674687562495
Since the home computer
only has 8 GB of RAM...8.5 billion points...Since not everybody has
a supercomputer reachable, a lot of efforts have been done on
processing massive data on even consumer hardware.
Here an example from a power user who managed to process 8.5 billion lidar points on his home computer.
With this touching base I would like to conclude my quick ride through 30 years of Open Source GIS development and conclude with...
Markus Neteler
Fondazione E. Mach (FEM)Centro Ricerca e InnovazioneGIS and Remote Sensing Unit38010 S. Michele all'Adige (Trento), Italyhttp://gis.cri.fmach.ithttp://www.osgeo.org
[email protected]@osgeo.org
Concluding remark:
THANKS TO ALL CONTRIBUTORS!
And more to come...
PostGISomics
Happy birthday, GRASS GIS
And
Thanks to all FOSS4G contributors all over the world.
Thank you for listening
Backup slides following
THE 90's ...
http://gis.cri.fmach.it/markus/grass_web_evolution/
FOSS Software development structures
Organizational structures of development teams
Developers
Head
Power users
Linux Kernel
GRASS Development Team/
GRASS Project Steering Committee
Decision level
+
-
SVN write
access
patch/docscontribution
Developers
User community
GRASS: No true BDFL (Benevolent
Dictator For Life)
http://en.wikipedia.org/wiki/Benevolent_Dictator_for_Life
http://producingoss.com/
Since 2010... new cool stuff
GIS on USB sticks
Mobile GIS
GIS today... beyond Desktop GIS
GIS Cloud computing
WebGIS
Since 2010...
Since 2010...
Markus Neteler 2013, CC-BY-SA