[Gnewsense-dev] Thoughts/questions on Builder and Dunsink

[Bob Ham]

Hi all,

I've had a look at Builder now and I thought it would be worth noting
some thoughts and questions:


0. A lot of the environment variables used as configuration options are
basically static data that describe possibilities in the structure of
the upstream archive. I think a better way for the scripts to work
would be to have a set of static data for different upstreams and
generalised scripts configured with a single option. For example, a
directory "upstreams/" containing "hardy.conf", "wheezy.conf",
"stretch.conf", etc. and a configuration option like "UPSTREAM=stretch".


1. Calls to the package-specific generation scripts are hard-coded.
Rather than having a long list of lines of, for example[0]

"ensure_updated blender blender $RELEASE$distro_release ./gen-blender $BLENDER_VERSION"

I think it would be better to have the package-specific generation
scripts detected automatically, along with any upstream- and
version-specific tweaks. For example, a directory "package-scripts/"
containing files in sub-directories like "blender/gen.sh",
"blender/upstreams/jessie/gen.sh" and possibly some common code to
automatically execute specific tweak scripts according to comparisons of
the package version or other variables.


2. There is a directory in bazaar named "packages-ucclia"[1].
Presumably this contains all of the packages for Ucclia. Are there any
gNewSense-built packages in Ucclia whose sources are not present in that
directory? Similarly, are there any changes in gNewSense-built packages
which are not listed in the bazaar history for the package's source
directory?


3. Sam, previously you stated that you would like to target Ucclia
(based on Wheezy) with new Builder scripts before moving on to Dunsink.
However, there are changes in the builder_lordeddi branch in bazaar
which refer to Debian Jessie[2] which would obviously not be targetting
Ucclia. Has there been a change in the intention to target Ucclia
first?


4. Also, to me it makes more sense to target Stretch rather than Jessie
for Dunsink. Stretch is already in "transition freeze" and will be
fully frozen this coming February. I can't imagine Builder being in a
place to create Ucclia, let alone Dunsink by then. Meanwhile, Jessie is
getting on for two years old.

The freeze period for Stretch would provide an appropriate period for
developing Dunsink. Having a Stretch-derived Dunsink released in tandem
with Stretch, or as close to as possible, would really be a boon to (1)
gNewSense's usefulness and appeal, and (2) the power of the statement
made by having a free version of Stretch. Sam, is it your intention for
Dunsink to be derived from Jessie?


5. In general, there seems to be quite a bit of technical debt. I've
noted a lot of copy-and-pasting, different indentation schemes within
the same script and even the same function, and so on. There's a fair
amount of cleaning up needed. Is there an explicit coding style noted
anywhere?


6. I discovered that Builder copies and makes use of Debian's .deb files
without recompilation. I was under the impression that all packages in
the archive would be recompiled. In fact I was under the impression
that recompilation was expected of all Debian derivatives but I looked
at Debian's guidelines and they state "for those derivatives that re-use
Debian binary packages"[3] so plainly they're OK with it. I believe
this[4] article gave me the wrong impression. I note this for anybody
following who may also be under the wrong impression.


Regards,

Bob


[0] http://bzr.savannah.gnu.org/lh/gnewsense/builder/annotate/head:/do-update#L337
[1] http://bzr.savannah.gnu.org/lh/gnewsense/packages-ucclia/
[2] http://bzr.savannah.gnu.org/lh/gnewsense/builder_lordeddi/revision/518
[3] https://wiki.debian.org/Derivatives/Guidelines#Repositories
[4] https://lwn.net/Articles/676664/

[Sam Geeraerts]

Op Tue, 06 Dec 2016 13:49:32 +0000

Post by Bob Ham
0. A lot of the environment variables used as configuration options
are basically static data that describe possibilities in the
structure of the upstream archive. I think a better way for the
scripts to work would be to have a set of static data for different
upstreams and generalised scripts configured with a single option.
For example, a directory "upstreams/" containing "hardy.conf",
"wheezy.conf", "stretch.conf", etc. and a configuration option like
"UPSTREAM=stretch".

Patches welcome.

Post by Bob Ham
1. Calls to the package-specific generation scripts are hard-coded.
Rather than having a long list of lines of, for example[0]
"ensure_updated blender blender $RELEASE$distro_release ./gen-blender $BLENDER_VERSION"
I think it would be better to have the package-specific generation
scripts detected automatically, along with any upstream- and
version-specific tweaks. For example, a directory "package-scripts/"
containing files in sub-directories like "blender/gen.sh",
"blender/upstreams/jessie/gen.sh" and possibly some common code to
automatically execute specific tweak scripts according to comparisons
of the package version or other variables.

There are dependencies between packages, e.g. some package builds query
dpkg-vendor, so calling generation scripts in a specific order is
useful. OTOH, packages are usually not built all at once, so there's
not that much of a chance that something goes wrong. Technically, a
better way to catch this issue is to flag all build-depending packages
when a package gets updated.

Again, patches welcome for you suggestion and mine. :)

Post by Bob Ham
2. There is a directory in bazaar named "packages-ucclia"[1].
Presumably this contains all of the packages for Ucclia. Are there
any gNewSense-built packages in Ucclia whose sources are not present
in that directory?

No.

Post by Bob Ham
Similarly, are there any changes in gNewSense-built packages which
are not listed in the bazaar history for the package's source
directory?

No.

Post by Bob Ham
3. Sam, previously you stated that you would like to target Ucclia
(based on Wheezy) with new Builder scripts before moving on to
Dunsink. However, there are changes in the builder_lordeddi branch in
bazaar which refer to Debian Jessie[2] which would obviously not be
targetting Ucclia. Has there been a change in the intention to
target Ucclia first?

Eddi convinced me (well, still need to do some more testing) that
merging current Ucclia's repo with Dunsink's repo is doable, so then
there's no need to reimplement Ucclia in Builder.

Post by Bob Ham
4. Also, to me it makes more sense to target Stretch rather than
Jessie for Dunsink. Stretch is already in "transition freeze" and
will be fully frozen this coming February. I can't imagine Builder
being in a place to create Ucclia, let alone Dunsink by then.
Meanwhile, Jessie is getting on for two years old.
The freeze period for Stretch would provide an appropriate period for
developing Dunsink. Having a Stretch-derived Dunsink released in
tandem with Stretch, or as close to as possible, would really be a
boon to (1) gNewSense's usefulness and appeal, and (2) the power of
the statement made by having a free version of Stretch. Sam, is it
your intention for Dunsink to be derived from Jessie?

Yes, unless you can convince me that we can have a smooth upgrade path
from Ucclia to a Stretch-derived Dunsink.

Post by Bob Ham
5. In general, there seems to be quite a bit of technical debt. I've
noted a lot of copy-and-pasting, different indentation schemes within
the same script and even the same function, and so on. There's a fair
amount of cleaning up needed. Is there an explicit coding style noted
anywhere?

No. Suggestions for coding style and lint tools/configurations are
welcome. But let's not get forget that style is a means to an end.

Post by Bob Ham
6. I discovered that Builder copies and makes use of Debian's .deb
files without recompilation. I was under the impression that all
packages in the archive would be recompiled. In fact I was under the
impression that recompilation was expected of all Debian derivatives
but I looked at Debian's guidelines and they state "for those
derivatives that re-use Debian binary packages"[3] so plainly they're
OK with it. I believe this[4] article gave me the wrong impression.
I note this for anybody following who may also be under the wrong
impression.

Rebuilding everything takes a *lot* of resources. It would fix minor
issues, e.g. distro info in Apache. It could be that our patches break
the builds of packages that we don't rebuild. Tracking build-depends
would catch that, but I suspect we rarely ever break a build.

[Bob Ham]

Post by Sam Geeraerts
Op Tue, 06 Dec 2016 13:49:32 +0000

Post by Bob Ham
3. Sam, previously you stated that you would like to target Ucclia
(based on Wheezy) with new Builder scripts before moving on to
Dunsink. However, there are changes in the builder_lordeddi branch in
bazaar which refer to Debian Jessie[2] which would obviously not be
targetting Ucclia. Has there been a change in the intention to
target Ucclia first?

Eddi convinced me (well, still need to do some more testing) that
merging current Ucclia's repo with Dunsink's repo is doable, so then
there's no need to reimplement Ucclia in Builder.

That's not cool. I have tens of GBs of Wheezy mirror on my hard disk
and a bunch of modifications to Builder to get it to build a
Wheezy-based distro. What a waste of time and bandwidth. And in a
project like gNewSense, where there is extremely limited manpower,
wasting time is disastrous.

I haven't seen any mention of a change in plans on the mailing list so I
can only conclude that this convincing you mention took place in
private. So I'm wasting my time because you're having conversations
about gNewSense development in private rather than on the development
mailing list. This is not the way to encourage developers to
contribute.

Are there any other development decisions you've made in private that I
should be aware of before I waste more time on pointless patches?

[Sam Geeraerts]

Op Mon, 12 Dec 2016 20:15:19 +0000

Post by Bob Ham
Are there any other development decisions you've made in private that
I should be aware of before I waste more time on pointless patches?

You're right that I said something about this on the mailing list.

In my previous mail I meant that there's a good chance of success that
they can be merged, not that it's definitive. As stated on the project
page [1], we first need to have a solution for merging distros, either
Builder-only ones or Builder + non-Builder ones. The former could be
integrated in Builder, the latter would be useful to merge Ucclia as-is.

I didn't know you were actively hacking on this. I didn't see a
buider_<username> bazaar repo on Savannah. I suspect you modified
Builder to make it more generic or better fit our current situation
(Debian instead of Ubuntu). We need to have those changes anyway, so
perhaps I can already merge a large part of your patches. I'm sure Eddi
will also be interested to compare with his changes. Can you please
share your changes?

[1] http://gnewsense.org/DevelopmentProjects/Gnewsense5

[Bob Ham]

Post by Sam Geeraerts
Op Mon, 12 Dec 2016 20:15:19 +0000

Post by Bob Ham
Are there any other development decisions you've made in private that
I should be aware of before I waste more time on pointless patches?

You're right that I said something about this on the mailing list.

I'm confused. This doesn't make sense to me. From what I can tell you
*didn't* say something about this.

Post by Sam Geeraerts
In my previous mail I meant that there's a good chance of success that
they can be merged, not that it's definitive.

It would be useful if you could be much louder about your decision
making; both the process and the outcome.

Post by Sam Geeraerts
As stated on the project
page [1], we first need to have a solution for merging distros

What exactly is it that you mean by "merging" distros here?

Post by Sam Geeraerts
I didn't know you were actively hacking on this. I didn't see a
buider_<username> bazaar repo on Savannah.

I didn't have anything working to commit. I still don't.

Post by Sam Geeraerts
I suspect you modified
Builder to make it more generic or better fit our current situation
(Debian instead of Ubuntu).

Your suspicion is not correct. My aim was to create something that
successfully created a Wheezy-based distro, which could be used as a
base for further work.

Post by Sam Geeraerts
We need to have those changes anyway, so
perhaps I can already merge a large part of your patches.

I doubt it; there is only one ugly patch. Regardless, this does not
change the fact that I've spent time working towards a goal which you
changed in private.

I'd like to be very clear and explicit about this issue. From my
perspective, making decisions about gNewSense development in private
discussion with Eddi is a problem. If one's behaviour reflects a belief
that there are no developers working on a project, for example being
quiet about major project decisions, then one will inevitably find this
to be a self-fulfilling prophesy and that, indeed, no developers work on
a project.

On the other hand, if one is loud and communicates in a manner that
assumes there are developers prepared to work on a project and in a
manner that facilitates an easy transition into being involved, then one
will attract developers.

Making decisions in private with Eddi is a problem in general. There
is, here, a specific example of why it is a problem because a developer
on the mailing list, me, has spent time working towards a goal whose
status has been changed in private while the developer continued
working.

Making decisions in private with Eddi is a specific problem in this case
but also a problem in general. Do you agree, Sam?

Post by Sam Geeraerts
I'm sure Eddi
will also be interested to compare with his changes. Can you please
share your changes?

I've created a branch here:

http://bzr.savannah.gnu.org/lh/gnewsense/builder_rah_wheezy/files

However, it is not working.

[Sam Geeraerts]

Op Fri, 30 Dec 2016 22:33:45 +0000

Post by Bob Ham

Post by Sam Geeraerts
You're right that I said something about this on the mailing list.

I'm confused. This doesn't make sense to me. From what I can tell
you *didn't* say something about this.

Argh, typing too fast: "... that I should have said something ...".

Post by Bob Ham
It would be useful if you could be much louder about your decision
making; both the process and the outcome.

I'll make it a new year's resolution.

It's also useful to check with the community if you're on the right
track when you're looking to contribute. If assumptions are made (e.g.
specialised vs. generic solution) it could be that you spend time on
something that's unlikely to be merged.

Post by Bob Ham

Post by Sam Geeraerts
As stated on the project
page [1], we first need to have a solution for merging distros

What exactly is it that you mean by "merging" distros here?

Combining /path/to/ucclia and /path/to/dunsink into 1 folder and still
have a working repo.

Post by Bob Ham

Post by Sam Geeraerts
I suspect you modified
Builder to make it more generic or better fit our current situation
(Debian instead of Ubuntu).

Your suspicion is not correct. My aim was to create something that
successfully created a Wheezy-based distro, which could be used as a
base for further work.

My suspicion doesn't exclude your aim. My assumption is that the work
for Dunsink would also be most of the work (barring specific gen
scripts) for future releases.

Post by Bob Ham
Making decisions in private with Eddi is a specific problem in this
case but also a problem in general. Do you agree, Sam?

Yes, I believe we've already established that.

Post by Bob Ham
http://bzr.savannah.gnu.org/lh/gnewsense/builder_rah_wheezy/files
However, it is not working.

Thanks, I'll have a look.

[l***@riseup.net]

Hi Bob, Sam,

I have been very busy, sorry for my lack of activity on this list.
I did not find the time yet to continue working on builder, I am
studying for my exams.
Starting February there should be more time to work on this.

Post by Sam Geeraerts

Post by Bob Ham
It would be useful if you could be much louder about your decision
making; both the process and the outcome.

I'll make it a new year's resolution.

I will not discuss gNewSense related things with Sam in private anymore.
It never was my intention to make descisions in private, nor to create
problems for anyone. I just wanted to learn and ask a batch of questions.
I am really sorry. I am learning from my mistakes.



Happy 2017

eddi

[Sam Geeraerts]

Op Sun, 1 Jan 2017 22:49:22 +0100

Post by Sam Geeraerts
Op Fri, 30 Dec 2016 22:33:45 +0000

Post by Bob Ham
http://bzr.savannah.gnu.org/lh/gnewsense/builder_rah_wheezy/files
However, it is not working.

Thanks, I'll have a look.

Bob, your changes look pretty good so far. What's your rationale for
making the inclusion of -security optional?

[Bob Ham]

Post by Sam Geeraerts
Op Sun, 1 Jan 2017 22:49:22 +0100

Post by Sam Geeraerts
Op Fri, 30 Dec 2016 22:33:45 +0000
http://bzr.savannah.gnu.org/lh/gnewsense/builder_rah_wheezy/files

Post by Bob Ham
However, it is not working.

Thanks, I'll have a look.

Bob, your changes look pretty good so far. What's your rationale for
making the inclusion of -security optional?

There were problems with -security in Wheezy. I forget now exactly
what the issue was. I thought it was because there was no -security in
Wheezy but looking at security.debian.org, there does seem to be a
wheezy dist there.

[l***@riseup.net]

Hi


these are the respective debmirror flags i used (for wheezy)
note the path difference under the --dist flag

- for debian
--host=ftp.be.debian.org
--dist=wheezy,wheezy-updates,wheezy-backports
--section=main,main/debian-installer,universe,contrib
--root=debian

- for debian-security
--host=security.debian.org
--dist=wheezy/updates
--section=main,main/debian-installer,universe,contrib
--root=debian-security

eddi

Post by Sam Geeraerts
Op Sun, 1 Jan 2017 22:49:22 +0100

Post by Sam Geeraerts
Op Fri, 30 Dec 2016 22:33:45 +0000

Post by Bob Ham
http://bzr.savannah.gnu.org/lh/gnewsense/builder_rah_wheezy/files
However, it is not working.

Thanks, I'll have a look.

Bob, your changes look pretty good so far. What's your rationale for
making the inclusion of -security optional?

There were problems with -security in Wheezy. I forget now exactly what
the issue was. I thought it was because there was no -security in
Wheezy but looking at security.debian.org, there does seem to be a
wheezy dist there.