A collaboration between Unitec and NIWA is turning big data into useful information and giving computing students the chance to work with real information.
To help make sense of the ever increasing volume of information NIWA receives from its monitoring it has formed a partnership with Unitec’s computing department to utilise its data analysis expertise.
Department of Computing’s Professor Paul Pang specialises in machine learning and creating algorithms to understand large data sets. He approached NIWA three years ago to offer his skills in exchange for access to its environmental data so he could test his work on real world information. This has led to a fruitful relationship which has seen important steps made in several areas, most significantly in the monitoring and modelling of air quality.
By taking data from a specialist monitoring machine used by NIWA, called PACMAN, Pang and his team have been able to get closer to identifying and understanding the huge amounts of environmental data provided by NIWA. PACMAN was designed to assess air pollutants inside a home, observing things like smoking, cooking and heating, but NIWA also has large data sets from their monitoring of outside pollutants such as vehicle exhausts and factory emissions.
NIWA Programme Leader for Atmospheric Environment, Health and Society Ian Longley says the relationship came at just the right time for NIWA. “In many ways it was very well timed,” he says. “I was coming to the realisation - as are many people in air quality science - that the amount of data we can collect is rapidly increasing. We’re entering an era of big data, and while some fields have the computing power to know what to do with that, we don’t.”
Pang says being able to work with NIWA and real data is a huge boost. “Computational environmental analysis is quite a hot topic in my field, and environmental science is the research direction of New Zealand,” he says. “But you need data. We believe that if we can work with industry partners, we’re applying real problem solving, which is better for us and our students. Then this system will be acknowledged by our industry partners, and they in turn will have a big contribution.”
Through the association with Longley, Pang has also been able to offer his expertise to other projects within NIWA. He is doing boat-flow analysis around the harbour areas, based on camera data collected by Ministry of Primary Industries. “They want to know about the numbers of boats in and out, to help assess the total amount in terms of the fishery,” says Pang.
Another project is around analysing scampi distribution. “They do surveys to find out how many burrows are made by scampi at the bottom of the sea, so they know their distribution,” says Pang. “Again, we have the data in the form of cameras, and they want us to create computing methods to count the scampi burrows, which will automatically estimate the distribution of scampi.”
Find out more information about the work with NIWA and the full article here.