SimpleVeggieGarden - roadblocks on a simple gps2svg bash script

This post is about the birth of SimpleVeggieGarden (a small bashscript I did to draw my garden), the challenges encountered, and how I (often) managed to bypass them.
Get the code on bitbucket

1. Introduction

It all started, when I wanted to get an overview the plants and trees of a plot of land.
I couldn't find what I was looking for in existing (open source) apps: here where my issues:
- Image editors are easy to start with, but it gets really hard to measure things (I don't draw well) and hard to manage when land complexity raises.
- Garden editors didn't fit my needs either, as they use weird (proprietary) formats and are missing a lot of plant taxons.
- Dia, like Gimp, got complex to manage when land complexity raised.

Hence I decided to start from scratch - as usual, using bash and limiting dependencies as much as possible (KISS).
The base idea is to store the plants and their coordinates in a database (mariadb/mysql), then generate an svg (txt) file based on that data.

It took me:
1. a couple of days to get the first prototype running.
2. a couple of weeks to implement database structure and modular script design
3. a couple of months to try to figure a way of getting decent datapoints at a reasonable price

2. Base prototype

The base concept, while least expecting it!
I had tried on and off to draw my garden in different formats for a few months, then spent a few weeks on Advent of Code, then went back to drawing - and I guess the AOC challenges made me see the problem from another perspective.
Instead of measuring my garden, the space between plants, ... why not generate a view based on GPS coordinates - some kind of wrapper that would translate lon/lat to svg coordinates?
I started by writing f_size_scale; which gets the min/max lat/lon from all points - and defines the size of the plane we are drawing.
Once this was done, all I had to do was adding a table "trees" - where each tree has it's GPS coordinates - and parse it through the coordinate wrapper.
This worked straight on, so I added another table to keep some "poly" (either lines or polygons (paths or areas)). With this, I have all the required tools to draw what I want: power lines, culture zones, walking paths, ...
All in all, in this part of development, I encountered two little challenges:
- Drawing a plane sphere.
While lattitude has a constant length/degree, longitude does not. A degree of longitude is a bit over 111 km at the equater, and 1m very close to the the poles: longitude scale depends on lattitude.
The scaling function was the first part into getting the concept working. It gets all gps coordinates from the project, keep the most east, west, north and south one - and de fine this area as our virtual grid.
It then takes the lattitude at the middle of min and max as base to define longitude scale.
- The Axis clusterfuck:
The first prototype kind of worked, but the whole drawing was rotated by 90° clockwise.
It took me a few hours to find it was the combination of 2 "stupid" mistakes:
- The first one was to use lat as x and lon as y on my graph - when in fact lat should be y and lon x (this causes a flip around an x=y axis)!
- The second mistake is that in gps coordinates, raising lat goes up (north) - while SVG is defined from top to bottom (raising lat values go "south"). - causing vertical flip.
Once those where corrected, the graph was drawn correctly according to the database.

3. Database Structure

Now that I had the what I needed to draw the data, all that was left was input some data right?
One point that seemed impossible to overcome was to encode relevant plant data.
Then I thought of basing my prototype on existing, scientific databases! Eureka!
A week later taxon_db.sh was born, importing data from the french 'Museum d'histoire Naturelle' and from the canadian 'canadensis'. This gave me a solid start to fill my tables.
As the french DB also recensed animals, bacteria, fungi, ... I included these too. This allows to declare relations (for example that a certain tree is affected by a specific mushroom, or hosts a specific insect). This could also allow for mapping animal observations.
As I now had names in latin/french/english for many plants, I added a table to contain language strings (table UI). This comes in very handy, as it allows for the generation of maps in the language of my guests
I did a few changes in the code so that the script could manage different projects, hence we have shared table for the UI, taxons and specs, and a set of tables for each project.

- UI: table with language localisation
- taxon_Animalia, taxon_Archaea, taxon_Bacteria, taxon_Chromista, taxon_Fungi, taxon_Plantae, taxon_Protozoa: tables with all taxons (species)
- specs_habitats, specs_rangs, specs_relations: referred to by taxon tables

- project1_cultures: culture zones: used to represent groups of plants - must be linked to a poly of type culture: IE: this zone is strawberries (cause you don't want to declare those one by one). Other exampe: this zone contains tomatoes, basil, onions and salads.
- project1_events: an event is linked to a culture or a tree (and a date) :examples: plant seed, harvest carrots, prune this tree)
- project1_poly: used to define custom zones (culture zone, lake, construction, ...) and lines (paths, electrical cables, irrigation pipes, ...)
- project1_relations: used to indicate relations between taxon tables (tree id 23 has a relation with mushroom 65)
- project1_trees: used to map individual plants. This table also includes data like the prune and harvest period - which will allow for "task maps" to be generated.

- project2_cultures project2_events project2_poly project2_relations project2_trees ==> same tables, different project
- mygarden_cultures mygarden_events mygarden_poly mygarden_relations mygarden_trees ==> same tables, different project

4. Getting the GPS data

Now that I could store and draw the data (with a fake test dataset), all that was left was to input real data to draw my garden.
I expected this to be the easiest part, I could not have been more wrong... Here is a list of what I have tried so far:

4.1. Simple test using the phone GPS

So I took a walk in the garden with my mobile, and mapped 20 trees. In theory this is working quite well!
In practice, I managed to get a precision of +- 4 meters on the phone - so the dot very often ends up at the wrong place - making this method completely unreliable.

To illustrate the issue:
Lets imagine the example of 6 trees, 1m in diameter, placed in a line, 1 meter apart from each other.
The green dots represent the trees, the blue error, the accuracy zone, the orange dots a possible GPS measurment.
- GPS has 4m accuracy

- GPS has 2m accuracy

- GPS has 1m accuracy

- GPS has 70cm accuracy

As we can see, 4m accuracy is completely useless, orange dots are not remotely near the green ones!
While the 2m accuracy looks better, it is still far from not really useable.
The 1m accuracy starts to be useable, while the 70cm really starts to be close (because 70cm is smaller than the 1m tree). This is the least scale of precision I'm aiming for.

4.2. Getting the coordinates from Google Maps

When something does not work, try Google right?
Zoom in on satelite view from google maps, right click tree, and bouya. Google maps gives you plenty of decimals this way (on computer, not on mobile).
In practice, this works very well for large plants which have been there for quite some time.
Small plants don't appear (map resolution too low), and trees you plant can take over 2 years to show up (google maps refresh rate) - leaving a big delay between planting and mapping - making this method unpractical (after two years you don't remember what you planted, and the little paper with the plant ID has disappeared a long time ago).

4.3. Layering drone capture with Google Maps

Then I thought about taking a picture of the land from above with a drone and open it in full screen.
Then open a browser on google maps with some transparency, and scale/align both images. Coordinates can then been obtained by clicking the Google maps.
Tried it: if I take the whole land, I miss resolution for the small plants. If I take closer shots, then perspective differences with the google maps pictures get amplified - and I must find the right zoom setting so that the image aligns with Google maps. It works... but takes sooo much time that it is not worth it,

4.4. Professional equipment (L2C)

Some people came home to mark land separation with the neighbours, they had a nice GPS - which gave them near centimeter accuracy.
Upon further research, I discovered that the model they used is "dual band" (L2C), which gives higher accuracy, more details: to-l2c-or-not-to-l2c-that-is-the-operational-question/
While this solution would definitely solve my precision issue, I am not planning to spend thousands of euros for this kind of apparatus.

4.5. Trying various phones

I then thought that I might try if any phones I could have access to would maybe have higher accuracy than the others...
At this point I also found out about a hidden developper option in Android, allowing for constant GNSS updates - enabled it in all tested phones ;)
--> I tried various phones at the same spot, and let them take some measurments (simultaneously).
- CAT S61, Redmi Note 9 pro, Moto g9 power, Doogee S96 Pro, Samsung Galaxy A50
All tested phones had accuracy between 4 and 8 meters, except the Doogee, which stood out around 2m! This was a big surprise as it is the only "unknown chinese brand" tested - now don't misread me, ... I don't especially recommend you buy a Doogee - I use it only in test situations.
- Example 2: Phone GPS has 2m accuracy
As you can see, this get much better than the 4m scenario - but still far from acceptable!

4.6. Aggregating measurments

One late evening, I had another idea... if the numbers are are wrong... maybe I could correct them with maths!
What if instead of taking one GPS reading per plant, I take many, and then run an average... The coordinates would then be statistically better.
But I must not be the first one to think of this... lets search the android store! After trying a few apps which I won't name I found GPS Point which did almost EXACTLY what I wanted (except it misses one decimal :'().
While device accuracy remained the same, precision of results increased quite a bit!

4.7. Dual frequency? L5?

Last week, I read this great article by Jack Price on XDA and realised that L5 might increase the precision I got from GPS (actually GNSS)!
The article mentions the BCM47755 chipset, but research taught me that since then, Broadcom released BCM47765.
I found sources mentioning Xiaomi mi 8 was using BCM47755, but nothing regarding BCM47765. Phone GPS specs mi8 and mi11, seem to indicate that in 11, xiaomi used BCM47765.
So I am now waiting for delivery of a Mi11 to see if might be the key to finally unlock the precision I have been looking for all these months.

4.8. Testing dual frequency with Xiaomi Mi 11

So I got the Mi 11, and as expected, results are unexptected - confusing at best!
The Mi 11 get much more satellites out of much more constellations than the other phones, including some L5, as expected with this chipset.
So I mapped a few points on the doogee and the mi11, to compare results: sometimes results where quite close, sometimes they differed by 3 to 5 meters...
I took a longer measurment at the angle of a shed that appears on the map - this allowed me to get a better understanding of the results!

On the left, the screenshot of the Mi11, on the right the Doogee.
At first glance, the Doogee, seems much better indeed, as it says it is accurate to 1.6m - while the Mi11 is accurate to 8.66m.
When you know that the phones where both located at the lower left corner of the building, the coordinates given by the Mi11 where much closer. On the Doogee phone, not even one of the main clusters contains the exact point where phones where.
This post about Precision vs accuracy explains it well:
The Mi11 is quite accurate, but not very precise.
The Doogee, is quite precise, but not very accurate!
For my needs, using "GPS point", the Mi11 gives a better answer!
Note: PPP Wizlite, developed for the French space agency (CNES) is probably the most scientifically accurate app to get a location - but also the "hardest" to run - as it needs internet to connect with an external source for orbital corrections.
- PPP Wizlite on playstore. If you want to try it.
- website of PPP Wizard. For more info

4.9. GPS Convergeance

After writing all this, I stumbled upon Precise Point Positioning Using Dual-Frequency GNSS Observations on Smartphone, which indicates in table 3 (page 10), that it took nearly 2 hours for a Mi8 to get 20cm accuracy (while all my measurments lasted less than 10 minutes).
Now... if I take 2 hours per plant... then either I need an armada of phones and chargers, or I need to get an extension on my lifetime if I want to finish mapping what I planned to map.
Anyway, I want to get high accuracy on at least one point, for the sake of the achievment ... Yesterday, I did a 4 hour measurment with PPP wizlite, and got an accuracy of 74cm, with 64% of battery left. Today I did a 5 hour measurment with PPP wizlite, and got a 64cm accuracy, with 1% of battery left. I then compared both measurments, and they are over 2m apart - which leads me to believe that I won't be able to obtain real precision and accuracy with this mobile (Mi 11). I would like to purchase an external battery pack, and try to make a 12 hour test, to see which accuracy I could possibly reach. Now... taking 5 hour per tree to map the garden would be a full time job for the mapping... so I should try to define an optimal measuring time. To determine the optimal time per tree to get an acceptable accuracy: If it's 2.30 minutes per tree, then I could map 200 trees/day - if it's 5 minutes per tree, then I can map 100 trees/day,...
The downside of this method, is that the smaller the plant, the more they are - and the more accuracy you probably need. To draw lines of small plants, the best option would be to take exact measurments of the 2 end of the line, and then to algorithmycly space the n plants on the line (calculating coordinates)

5. Conclusion

I am happy that my script works, and that I might finally be able to draw my garden some day - I'm even happier that I was able to include "scientific accuracy", both for the taxons and GPS measurments.
As usual, the hard parts came where I least expected them, I had a lot of fun figuring how to make things work and putting it all together. Writing this tiny script required some learning, creativity and improvisation!
Tools over 100Mb wouldn't do what I needed, I got it under 100kb (including the taxon db import code)!
The current precision/accuracy that I get using GPS point on the Xiaomi Mi 11 is not perfect yet, but sufficient for me to give it a real life use!
What is important is that thanks to the tools and data provided by the Open Source and scientific communities, someone who is neither a programmer nor a scientist managed to transform an unsatisfied need into a fun opporunity to learn tead of frustration!
I really hope GPS point will get an extra decimal soon, as accuracy should rise with L5 availability.
I hope this SimpleVeggieGarden will be useful to someone!