[Gnewsense-dev] Current Status (and Wiki Performance)

[Paul Boddie]

Hello,

I was looking at various "libre" distributions a while ago, and gNewSense was
one I briefly evaluated:

http://lists.phcomp.co.uk/pipermail/arm-netbook/2016-July/011088.html

It seemed that there is now a certain amount of re-architecting going on to
make new gNewSense releases easier, but I don't see so much progress on the
more prominent sites associated with the project.

I'm interested in trying to build a minimal gNewSense archive for mipsel, but
I'm not sure where to start and what future work might need doing so that a
recent Debian release can be transformed for this purpose.

Another thing I noticed was that the project wiki is very slow. On that topic,
I can perhaps offer some advice and maybe some assistance if that is
desirable.

Thanks for reading!

Paul

[Sam Geeraerts]

Op Sun, 28 Aug 2016 19:03:29 +0200

Post by Paul Boddie
It seemed that there is now a certain amount of re-architecting going
on to make new gNewSense releases easier, but I don't see so much
progress on the more prominent sites associated with the project.

It's still in the planning phase. Also, I've been pushing the limits of
what my own machines can do, so I think it's that time of the decade
when I need to get some new(er) gear to do some real work.

Post by Paul Boddie
I'm interested in trying to build a minimal gNewSense archive for
mipsel, but I'm not sure where to start and what future work might
need doing so that a recent Debian release can be transformed for
this purpose.

Do you have a particular mipsel platform in mind?

Post by Paul Boddie
Another thing I noticed was that the project wiki is very slow. On
that topic, I can perhaps offer some advice and maybe some assistance
if that is desirable.

Known issue. It's been suggested to run MoinMoin on nginx, but I
couldn't get that running myself. Feel free to share your knowledge
about that. On the other hand, I've spent so much time on this without
progress that maybe I'd better just start from scratch with a fresh
server setup.

[Paul Boddie]

Post by Sam Geeraerts
Op Sun, 28 Aug 2016 19:03:29 +0200

Post by Paul Boddie
It seemed that there is now a certain amount of re-architecting going
on to make new gNewSense releases easier, but I don't see so much
progress on the more prominent sites associated with the project.

It's still in the planning phase. Also, I've been pushing the limits of
what my own machines can do, so I think it's that time of the decade
when I need to get some new(er) gear to do some real work.

I know the feeling!

Post by Sam Geeraerts

Post by Paul Boddie
I'm interested in trying to build a minimal gNewSense archive for
mipsel, but I'm not sure where to start and what future work might
need doing so that a recent Debian release can be transformed for
this purpose.

Do you have a particular mipsel platform in mind?

Well, to begin with, I'd target the Ben NanoNote, which is a fairly old system
now with limited performance but which does run Debian. I did hope to
experiment with the jz4775 computer card that looked as if it might be
delivered in the EOMA68 campaign, but that's not looking likely as a near-term
product.

https://www.crowdsupply.com/eoma68/micro-desktop
http://rhombus-tech.net/ingenic/jz4775/

We're talking about Ingenic SoCs here, with the Ben NanoNote already supported
in the Linux kernel (QI_LB60). I built the 4.7.2 release and it actually ran
on the Ben and my Debian installation worked with it, too. The people involved
have done a great job keeping support for the Ben up to date.

I'm not necessarily planning to get one, but the MIPS Creator CI20 uses a
jz4780 SoC and runs Debian. The GCW-Zero handheld games console (apparently
somewhat difficult to obtain) uses the jz4770. Once upon a time, there was a
mini-PC available with a variety of branding which uses the jz4730, but that
SoC is severely under-documented, although I do plan to get recent software
working on it some day.

Post by Sam Geeraerts

Post by Paul Boddie
Another thing I noticed was that the project wiki is very slow. On
that topic, I can perhaps offer some advice and maybe some assistance
if that is desirable.

Known issue. It's been suggested to run MoinMoin on nginx, but I
couldn't get that running myself. Feel free to share your knowledge
about that. On the other hand, I've spent so much time on this without
progress that maybe I'd better just start from scratch with a fresh
server setup.

I can easily check with various other Moin users to see which strategy works
best. Sometimes, just using mod_wsgi seems to work out. Getting to know the
nature of the load you're experiencing may be useful in improving the problem,
though.

I'm no deployment expert, but I can offer some experience on configuring Moin
to resist problematic and demanding usage.

Paul

[Sam Geeraerts]

Op Tue, 30 Aug 2016 23:22:07 +0200

Post by Paul Boddie
Well, to begin with, I'd target the Ben NanoNote, which is a fairly
old system now with limited performance but which does run Debian.

Cool. I have one of those lying around. I did run gNewSense on it once,
but then other stuff came up and it's been gathering dust for a while
now. It would be neat to have it running again.

Post by Paul Boddie
I did hope to experiment with the jz4775 computer card that looked as
if it might be delivered in the EOMA68 campaign, but that's not
looking likely as a near-term product.
https://www.crowdsupply.com/eoma68/micro-desktop
http://rhombus-tech.net/ingenic/jz4775/

Shame. It would be nice if we could have FSF endorsed hardware as a
build machine.

Post by Paul Boddie
We're talking about Ingenic SoCs here, with the Ben NanoNote already
supported in the Linux kernel (QI_LB60). I built the 4.7.2 release
and it actually ran on the Ben and my Debian installation worked with
it, too. The people involved have done a great job keeping support
for the Ben up to date.
I'm not necessarily planning to get one, but the MIPS Creator CI20
uses a jz4780 SoC and runs Debian. The GCW-Zero handheld games
console (apparently somewhat difficult to obtain) uses the jz4770.
Once upon a time, there was a mini-PC available with a variety of
branding which uses the jz4730, but that SoC is severely
under-documented, although I do plan to get recent software working
on it some day.

I was actually considering dropping mipsel support after Debian Jessie,
because Debian then no longer supports the Yeeloong [1]. But if there
are other mipsel machines that are worth supporting (and still
supported by Debian) then we might keep it. But then I'd need some
hardware that can build and run it. Maybe the CI20 is a potential build
machine?

Post by Paul Boddie
I can easily check with various other Moin users to see which
strategy works best. Sometimes, just using mod_wsgi seems to work
out. Getting to know the nature of the load you're experiencing may
be useful in improving the problem, though.

Indeed. In the beginning it was spambots. They're still at it, but not
all that heavy according to web server logs. So now I'm not sure.

Post by Paul Boddie
I'm no deployment expert, but I can offer some experience on
configuring Moin to resist problematic and demanding usage.

I appreciate any tips. But sometimes it responds fairly quickly, so I'm
not sure Moin configuration is the problem.

[1] https://lists.debian.org/debian-mips/2015/09/msg00023.html

[Paul Boddie]

Post by Sam Geeraerts
Op Tue, 30 Aug 2016 23:22:07 +0200

Post by Paul Boddie
Well, to begin with, I'd target the Ben NanoNote, which is a fairly
old system now with limited performance but which does run Debian.

Cool. I have one of those lying around. I did run gNewSense on it once,
but then other stuff came up and it's been gathering dust for a while
now. It would be neat to have it running again.

So I guess there were gNewSense builds for mipsel at some point to support the
Lemote products, and that these worked with the various kernels provided for
the Ben NanoNote?

Post by Sam Geeraerts

Post by Paul Boddie
I did hope to experiment with the jz4775 computer card that looked as
if it might be delivered in the EOMA68 campaign, but that's not
looking likely as a near-term product.
https://www.crowdsupply.com/eoma68/micro-desktop
http://rhombus-tech.net/ingenic/jz4775/

Shame. It would be nice if we could have FSF endorsed hardware as a
build machine.

Fortunately/unfortunately, it was shown that the A20 card can most likely be
endorsed, thus eliminating any urgent need for the jz4775 card. But that would
also mean that the A20 card could be used instead (to support ARM systems,
obviously).

[...]

Post by Sam Geeraerts
I was actually considering dropping mipsel support after Debian Jessie,
because Debian then no longer supports the Yeeloong [1]. But if there
are other mipsel machines that are worth supporting (and still
supported by Debian) then we might keep it. But then I'd need some
hardware that can build and run it. Maybe the CI20 is a potential build
machine?

The reference you provide [1] is somewhat confusing, partly because I don't
really know that much about the Yeeloong, but also because I thought that it
was a MIPS64 system. Maybe only the later models/variants are MIPS64, but it
then confuses me that this change would seek to affect MIPS32 systems, too.

Interestingly, Guix(SD) supports mips64el presumably for Yeeloong
compatibility, but I'm not too familiar with Guix at the moment, so I don't
know what architecture version restrictions that system imposes in its build
environment. I can always dig up some more information about that.

I guess the CI20 has some potential. It probably isn't the most powerful kind
of system, and I don't know what the jz4780 (used in the CI20) offers over the
jz4720 (used in the Ben NanoNote, which doesn't have hardware floating point).

In addition, the jz4780 uses widely-disliked PowerVR technologies, but maybe
they are not mandatory for it to be used in a productive way. And Imagination
Technologies seem to have moved on to the CI40 (which is less suitable as a
build host), and I don't see a particularly large amount of obvious support
for the CI20 any more.

Post by Sam Geeraerts

Post by Paul Boddie
I can easily check with various other Moin users to see which
strategy works best. Sometimes, just using mod_wsgi seems to work
out. Getting to know the nature of the load you're experiencing may
be useful in improving the problem, though.

Indeed. In the beginning it was spambots. They're still at it, but not
all that heavy according to web server logs. So now I'm not sure.

The last time I had a look at a widely-used public Moin instance with
potential spam problems, the biggest inconvenience was account creation spam,
but I think this was solved by adding a registration textcha. I don't think
this really affected performance, though: editing was restricted through the
use of editor groups and ACLs, which is possibly the safest configuration
these days with Moin.

Post by Sam Geeraerts

Post by Paul Boddie
I'm no deployment expert, but I can offer some experience on
configuring Moin to resist problematic and demanding usage.

I appreciate any tips. But sometimes it responds fairly quickly, so I'm
not sure Moin configuration is the problem.

Maybe it's some other kind of server issue. Anyway, if you need any assistance
then I'd be happy to take a look.

Paul

Post by Sam Geeraerts
[1] https://lists.debian.org/debian-mips/2015/09/msg00023.html

[Sam Geeraerts]

Op Thu, 1 Sep 2016 00:18:47 +0200

Post by Paul Boddie
So I guess there were gNewSense builds for mipsel at some point to
support the Lemote products, and that these worked with the various
kernels provided for the Ben NanoNote?

I don't remember if I used the gNewSense kernel or another one.

Post by Paul Boddie
Fortunately/unfortunately, it was shown that the A20 card can most
likely be endorsed, thus eliminating any urgent need for the jz4775
card. But that would also mean that the A20 card could be used
instead (to support ARM systems, obviously).

Hm, considering the problems we've had with mipsel, I'm not keen on
adding another architecture to the repo.

Post by Paul Boddie
The reference you provide [1] is somewhat confusing, partly because I
don't really know that much about the Yeeloong, but also because I
thought that it was a MIPS64 system. Maybe only the later
models/variants are MIPS64, but it then confuses me that this change
would seek to affect MIPS32 systems, too.

As I understand it, it can run 32 bit and 64 bit ABIs [1] and the "one
size fits all" MIPS I o32 instruction set that Debian provided, but not
the MIPS II instruction set to which they're switching now.

Post by Paul Boddie
Interestingly, Guix(SD) supports mips64el presumably for Yeeloong
compatibility, but I'm not too familiar with Guix at the moment, so I
don't know what architecture version restrictions that system imposes
in its build environment. I can always dig up some more information
about that.

It's good to know that there are other options that still support the
Yeeloong.

Post by Paul Boddie
In addition, the jz4780 uses widely-disliked PowerVR technologies,
but maybe they are not mandatory for it to be used in a productive
way. And Imagination Technologies seem to have moved on to the CI40
(which is less suitable as a build host), and I don't see a
particularly large amount of obvious support for the CI20 any more.

So that's a dead end, then. So if I understand you correctly, then with
regards to current and near-future hardware options, there's no real
reason to continue mipsel support in gNewSense post-Yeeloong.

Post by Paul Boddie
Maybe it's some other kind of server issue. Anyway, if you need any
assistance then I'd be happy to take a look.

Ack.

[1]
https://blogs.gentoo.org/blueness/2014/07/02/continued-support-for-the-lemote-yeeloong-gentoo-mips-is-alive-and-well/

[Paul Boddie]

Post by Sam Geeraerts
Op Thu, 1 Sep 2016 00:18:47 +0200

Post by Paul Boddie
So I guess there were gNewSense builds for mipsel at some point to
support the Lemote products, and that these worked with the various
kernels provided for the Ben NanoNote?

I don't remember if I used the gNewSense kernel or another one.

OK, but as I understand it, then, gNewSense must have been supporting mipsel
instead of mips64el and thus the distribution worked on the Ben. (My
impression is that mips64el is a fairly new thing in Debian.)

Post by Sam Geeraerts

Post by Paul Boddie
Fortunately/unfortunately, it was shown that the A20 card can most
likely be endorsed, thus eliminating any urgent need for the jz4775
card. But that would also mean that the A20 card could be used
instead (to support ARM systems, obviously).

Hm, considering the problems we've had with mipsel, I'm not keen on
adding another architecture to the repo.

Well, this was more a personal experiment at present. I'm not advocating
anything for the distribution itself.

Post by Sam Geeraerts

Post by Paul Boddie
The reference you provide [1] is somewhat confusing, partly because I
don't really know that much about the Yeeloong, but also because I
thought that it was a MIPS64 system. Maybe only the later
models/variants are MIPS64, but it then confuses me that this change
would seek to affect MIPS32 systems, too.

As I understand it, it can run 32 bit and 64 bit ABIs [1] and the "one
size fits all" MIPS I o32 instruction set that Debian provided, but not
the MIPS II instruction set to which they're switching now.

I think the Ingenic SoCs may support MIPS II, but I'm not entirely sure, and
their datasheets probably intentionally do not indicate this because the
"XBurst" architecture is based on presumably-unencumbered aspects of the MIPS
architecture. They may have licensed the architecture from ImgTec
subsequently, however.

Post by Sam Geeraerts

Post by Paul Boddie
Interestingly, Guix(SD) supports mips64el presumably for Yeeloong
compatibility, but I'm not too familiar with Guix at the moment, so I
don't know what architecture version restrictions that system imposes
in its build environment. I can always dig up some more information
about that.

It's good to know that there are other options that still support the
Yeeloong.

Yes.

Post by Sam Geeraerts

Post by Paul Boddie
In addition, the jz4780 uses widely-disliked PowerVR technologies,
but maybe they are not mandatory for it to be used in a productive
way. And Imagination Technologies seem to have moved on to the CI40
(which is less suitable as a build host), and I don't see a
particularly large amount of obvious support for the CI20 any more.

So that's a dead end, then. So if I understand you correctly, then with
regards to current and near-future hardware options, there's no real
reason to continue mipsel support in gNewSense post-Yeeloong.

I don't know. The Ingenic SoCs have been used in a variety of devices, but I'm
not sure how available those devices are now. The GCW-Zero is perhaps the most
well-known, but it seems to suffer availability problems, although there does
seem to be some kind of community around it and other devices.

Post by Sam Geeraerts

Post by Paul Boddie
Maybe it's some other kind of server issue. Anyway, if you need any
assistance then I'd be happy to take a look.

Ack.
[1]
https://blogs.gentoo.org/blueness/2014/07/02/continued-support-for-the-lemo
te-yeeloong-gentoo-mips-is-alive-and-well/

Thanks for the references!

Where should I start looking if I want to derive a gNewSense variant from
Debian for the purposes of evaluating the feasibility of my little project? I
did find this...

http://bzr.savannah.gnu.org/lh/gnewsense/

...but it seems to be down at present.

Paul

[Sam Geeraerts]

Op Sat, 3 Sep 2016 00:24:18 +0200

Post by Paul Boddie
OK, but as I understand it, then, gNewSense must have been supporting
mipsel instead of mips64el and thus the distribution worked on the
Ben. (My impression is that mips64el is a fairly new thing in Debian.)

Indeed.

Post by Paul Boddie
I don't know. The Ingenic SoCs have been used in a variety of
devices, but I'm not sure how available those devices are now. The
GCW-Zero is perhaps the most well-known, but it seems to suffer
availability problems, although there does seem to be some kind of
community around it and other devices.

I'm guessing they're not really our target audience.

Post by Paul Boddie
Where should I start looking if I want to derive a gNewSense variant
from Debian for the purposes of evaluating the feasibility of my
little project? I did find this...
http://bzr.savannah.gnu.org/lh/gnewsense/
...but it seems to be down at present.

bzr branch bzr://bzr.savannah.nongnu.org/gnewsense/debderiver/

That's what we use to derive from Debian. It's mostly a frontend for
reprepro. (Re-)building packages is done separately.

[Paul Boddie]

Sorry to revive an old thread, but I remembered mentioning something that I

Post by Sam Geeraerts

Post by Paul Boddie
In addition, the jz4780 uses widely-disliked PowerVR technologies,
but maybe they are not mandatory for it to be used in a productive
way. And Imagination Technologies seem to have moved on to the CI40
(which is less suitable as a build host), and I don't see a
particularly large amount of obvious support for the CI20 any more.

So that's a dead end, then. So if I understand you correctly, then with
regards to current and near-future hardware options, there's no real
reason to continue mipsel support in gNewSense post-Yeeloong.

I have since acquired a CI20 because I wanted a way of experimenting with the
Ingenic SoC family used in the Ben NanoNote, and it may also get pressed into
use as a general-purpose machine. Initial investigations indicate that you can
deploy Debian on it just fine without either of the proprietary firmware blobs
that are needed for the PowerVR-based GPU and Broadcom-derived WLAN functions.

One thing that came up in the context of the EOMA68 stuff is that the FSF did
seem to be considering endorsing hardware that has functions unsupportable by
Free Software if those functions are not then supported by proprietary
firmware, nor enabled in any way, nor even advertised in any way. This way,
such a device doesn't promote proprietary software and contradict the FSF's
goals.

Thus, the EOMA68 crowd-funding campaign was able, considering this tentative
position, to offer an Allwinner A20-based product on the basis that without
the Mali GPU functionality the rest of the offered functionality can be
supported by Free Software. Originally, the plan seemed to involve offering an
Ingenic jz4775-based card as the FSF-endorsed option instead.

So, maybe the CI20 is also endorseable in some sense according to these
broader criteria. It's just unfortunate that there doesn't seem to be a very
high level of corporate support for it any more, but you can still buy it from
Mouser and RS Online.

Paul

P.S. Personally, I think Luke (of the EOMA68 campaign) should have offered the
jz4775-based card that was already being prototyped instead, especially since
it has no known Free Software support issues and was already capable of
offering 2GB RAM, which is something since added to the A20 card, perhaps
complicating the process of getting the card made and shipped to the rather
patient group of backers.

[Sam Geeraerts]

Op Fri, 11 Aug 2017 15:52:29 +0200

Post by Paul Boddie

Post by Sam Geeraerts
So that's a dead end, then. So if I understand you correctly, then
with regards to current and near-future hardware options, there's
no real reason to continue mipsel support in gNewSense
post-Yeeloong.

So, maybe the CI20 is also endorseable in some sense according to
these broader criteria. It's just unfortunate that there doesn't seem
to be a very high level of corporate support for it any more, but you
can still buy it from Mouser and RS Online.

How viable is it? Is it worth spending effort to support it?

[Paul Boddie]

Post by Sam Geeraerts

Post by Paul Boddie

Post by Sam Geeraerts
So that's a dead end, then. So if I understand you correctly, then
with regards to current and near-future hardware options, there's
no real reason to continue mipsel support in gNewSense
post-Yeeloong.

So, maybe the CI20 is also endorseable in some sense according to
these broader criteria. It's just unfortunate that there doesn't seem
to be a very high level of corporate support for it any more, but you
can still buy it from Mouser and RS Online.

How viable is it? Is it worth spending effort to support it?

I'm doubtful. It would be useful if there were clear support from the
manufacturer or, failing that, an obvious progression of similar products, but
in recent times there were only the following (32-bit) mipsel-based products
using this SoC family that I was aware of:

GCW-Zero [1] - uses JZ4770 which has a Vivante GPU that might be supportable,
but getting hold of new ones appears to be a cat-and-mouse game. This is also
a handheld, so perhaps not the primary focus of gNewSense.

MIPS Creator CI20 [2] - uses JZ4780 which has an unsupportable PowerVR GPU,
although that and the Broadcom-based WLAN are perhaps optional, and at least
you can buy one. This is for "lightweight" desktop use, and arguments can be
had about competitiveness with things like the Raspberry Pi, but it does have
1GB RAM, unlike a lot of earlier single-board computers.

EOMA68-JZ4775 [3] - uses JZ4775 which has no GPU, but this hasn't been
anything more than a prototype. This is theoretically for desktop and laptop
use. The developer of this is currently trying to get an ARM-based product
done, and he was also looking at other ARM SoCs, so I'm starting to doubt that
we'll ever see this in the wild.

I don't really follow other MIPS-based products, but I did notice that the
GnuBee Personal Cloud 1 uses a MIPS-based MediaTek MT7621 CPU [4] and
apparently needs no proprietary firmware [5]. The MT7621 is a MIPS32r2 device
according to the OpenWrt support [6]. However, this product is a kind of
storage unit, not a display-connected device, and it only has 512MB RAM.

It's quite the coincidence that you ask about this right now because I was
just preparing another root filesystem for the CI20 to test Debian on it
again. The kernel is 3.18 and some mainline targeting was being done, although
I'm not sure how much corporate focus there is on this at the moment.

For me, a more serious issue is the weird interactions between lightdm and
DBus that prevented an evaluation of XFCE, so I'm going to try LXDE and see if
that works any better.

Hopefully, I'll be able to report back with some meaningful discoveries. Just
to see if the native performance is any good will be informative.

Paul

[1] http://www.gcw-zero.com/

[2] http://www.elinux.org/MIPS_Creator_CI20

[3] http://rhombus-tech.net/ingenic/jz4775/

[4] https://www.crowdsupply.com/gnubee/personal-cloud-1/updates/overclocked-
pre-production-units

[5] https://www.crowdsupply.com/gnubee/personal-cloud-1/updates/100-blob-free

[6] https://wiki.openwrt.org/doc/hardware/soc/soc.mediatek

[Bob Ham]

I asked on the #gnewsense-dev IRC channel but I didn't get any response
so I'll ask here.

Post by Sam Geeraerts
Op Sun, 28 Aug 2016 19:03:29 +0200

Post by Paul Boddie
It seemed that there is now a certain amount of re-architecting going
on to make new gNewSense releases easier, but I don't see so much
progress on the more prominent sites associated with the project.

It's still in the planning phase.

Could you possibly say more about the planning process and who's
involved? Is there anything holding up the planning? It's been five
months since a Builder-style approach was discussed but there doesn't
seem to have been any movement towards version 5 at all which is a bit
of a concern.

Have you decided whether to use a Builder-style approach yet? I'd like
to work on the scripts for such a system but obviously I don't want to
duplicate effort.

Bob

[l***@riseup.net]

Hi,

I emailed samgee a while ago about gNewSense Builder and have started
setting up a server following the documentation [1][2]

The main aim is to get the original Builder scripts [2] working for
Ucclia (wheezy) and move from there.

I am currently using ansible to setup my-ever-changing-environment,
the changes i made to the Builder scripts are also in there.
At the moment i only have a private bitbucket git and need todo some
refactoring.
Samgee recommended re-creating builder bzr repo and start clean, i want
to implement that first, then i can make it public.

I'm trying to figure out how all this works.
I'm a student and learning


i hope to take a look at what Trisquel [3] and devuan [4][5] are doing
also try using jenkins and look at other possible improvements.

Help on e.g. porting the ucclia changes to the gen-scripts, is welcome.
I have not gotten there yet.

eddi


[1] http://www.gnewsense.org/Builder
[2] http://www.gnewsense.org/Builder/HowToCreateYourOwnGNULinuxDistribution
[3] https://trisquel.info/en/wiki/how-trisquel-made
[4] https://devuan.org/
[5] https://ci.devuan.org/

Post by Bob Ham
I asked on the #gnewsense-dev IRC channel but I didn't get any response
so I'll ask here.

Post by Sam Geeraerts
Op Sun, 28 Aug 2016 19:03:29 +0200

Post by Paul Boddie
It seemed that there is now a certain amount of re-architecting going
on to make new gNewSense releases easier, but I don't see so much
progress on the more prominent sites associated with the project.

It's still in the planning phase.

Could you possibly say more about the planning process and who's
involved? Is there anything holding up the planning? It's been five
months since a Builder-style approach was discussed but there doesn't
seem to have been any movement towards version 5 at all which is a bit
of a concern.
Have you decided whether to use a Builder-style approach yet? I'd like
to work on the scripts for such a system but obviously I don't want to
duplicate effort.
Bob
_______________________________________________
gNewSense-dev mailing list
https://lists.nongnu.org/mailman/listinfo/gnewsense-dev

[Bob Ham]

Post by l***@riseup.net
I emailed samgee a while ago about gNewSense Builder and have started
setting up a server following the documentation [1][2]
The main aim is to get the original Builder scripts [2] working for
Ucclia (wheezy) and move from there.

My question was to Sam and regarded his planning for gNewSense 5. You
said you emailed Sam a while ago and yet at the end of August, Sam said
gNewSense 5 was still being planned. It's not clear whether your work
is intended to form the basis of gNewSense 5 in the minds of the
gNewSense developers.

Here are some questions it would be good to have clarity on:

- Who is working on what?
- Who are the people making decisions about gNewSense 5?
- Is Sam the only person making decisions about gNewSense 5?
- What are the decisions that need to be made for the planning of
gNewSense 5?
- Will gNewSense 5 use a Builder-style system?

Post by l***@riseup.net
I am currently using ansible to setup my-ever-changing-environment,
the changes i made to the Builder scripts are also in there.
At the moment i only have a private bitbucket git and need todo some
refactoring.
Samgee recommended re-creating builder bzr repo and start clean, i want
to implement that first, then i can make it public.
I'm trying to figure out how all this works.
I'm a student and learning

To be frank, it doesn't sound like what you've produced is going be the
basis of gNewSense 5. You said you want to reimplement a recreation of
Builder. That sounds like what could become the basis of gNewSense 5.
However, that doesn't exist yet.


Regarding the need to "refactor" your private things and making things
"public", I'd recommend reading The Cathedral and the Bazaar by Eric S.
Raymond:

http://www.catb.org/esr/writings/cathedral-bazaar/cathedral-bazaar/

If you feel your code is of such low quality that you're not even
prepared to let other people see it then I can't imagine it's going to
be used as the basis of gNewSense 5, which is the issue at hand.

Bob

[Sam Geeraerts]

Op Thu, 13 Oct 2016 09:40:27 +0100

Post by Bob Ham
- Who is working on what?

Eddi expressed interest in working on Builder as part of a school
project. I'm answering any questions he has.

Post by Bob Ham
- Who are the people making decisions about gNewSense 5?

Me.

Post by Bob Ham
- Is Sam the only person making decisions about gNewSense 5?

Yes.

Post by Bob Ham
- What are the decisions that need to be made for the planning of
gNewSense 5?
- Will gNewSense 5 use a Builder-style system?

I think Builder-style makes the most sense. We can work on that right
now. I want to keep Ucclia in the repo, so I'd like to port the old
Builder code to Ucclia first (so write gen script equivalents for
packages-ucclia), but it couldn't hurt to also get something for
Ucclia+1 going so that we can verify that we can combine multiple
distros in 1 repo. Solving the other cons [1] is a nice bonus that we
can focus on later.

Post by Bob Ham
If you feel your code is of such low quality that you're not even
prepared to let other people see it then I can't imagine it's going to
be used as the basis of gNewSense 5, which is the issue at hand.

Eddi picked this up as a potential school project. When you're getting
familiar with a new project it's understandable that you add some debug
code or hard coded stuff to figure things out, If you're working with
new tools and you're feeling your way around a community it makes sense
to be careful. Getting told that your code suck even before you show it
is also not a confidence booster.

I will talk about this further with Eddi. I hope we can publish
something soon and that this will be met with constructive criticism
from the community.

[1] http://www.gnewsense.org/DevelopmentProjects/Gnewsense5

[Paul Boddie]

Post by Sam Geeraerts
Eddi picked this up as a potential school project. When you're getting
familiar with a new project it's understandable that you add some debug
code or hard coded stuff to figure things out, If you're working with
new tools and you're feeling your way around a community it makes sense
to be careful. Getting told that your code suck even before you show it
is also not a confidence booster.

Agreed. It's difficult enough just to get a handle on distribution-building
processes. I've heard that the Devuan people have improved on Debian's
processes, but good luck finding out about such improvements! I think I may
have stumbled across something here:

https://git.devuan.org/devuan-infrastructure/amprolla

Or even better:

https://git.devuan.org/groups/devuan-infrastructure

It would be pretty useful in and of itself to just document what different
distributions actually do, given that many of them don't really see the need
(or don't do so concisely). That might help any future discussions about what
might work best.

(I found that Guix/GuixSD is one of the few distributions where the
documentation seeks to explain things in a coherent start-to-end fashion, but
even then it isn't a "how to", which is really what would be most
interesting.)

Post by Sam Geeraerts
I will talk about this further with Eddi. I hope we can publish
something soon and that this will be met with constructive criticism
from the community.

Indeed. I look forward to seeing Eddi's efforts.

Paul

[Bob Ham]

Post by Sam Geeraerts
Op Thu, 13 Oct 2016 09:40:27 +0100

Post by Bob Ham
- What are the decisions that need to be made for the planning of
gNewSense 5?

There's no response to this question. Should we assume that there are
no pending decisions which need to be made?

Post by Sam Geeraerts

Post by Bob Ham
- Will gNewSense 5 use a Builder-style system?

I think Builder-style makes the most sense. We can work on that right
now.

OK, so if there's a "we" and we are working together on gNewSense 5 then
we need to have awareness of what each person is working on and what is
happening in general. I've been waiting five months on the assumption
that you, Sam, were actually working on gNewSense 5 quietly. Of course,
this is my fault for making assumptions but I think moving forward, if
there is going to be effective team work, there needs to be much better
communication.

Post by Sam Geeraerts
I will talk about this further with Eddi.

Ordinarily, such conversations would happen in the open on a development
mailing list like this one. From the way you've phrased your sentence
and the fact that there has been no public discussion up to now, it
appears that you are intending to talk with Eddi privately. I'd like to
ask: is there a particular reason for holding these discussions in
private?

Post by Sam Geeraerts
I hope we can publish
something soon and that this will be met with constructive criticism
from the community.

I'm concerned by this mode of working. It seems that there is (Sam
+Eddi) doing work in private who will then seek feedback from some
separate entity called "the community". What I would expect, and what
is the norm in modern bazaar-style development is that Sam and Eddi are
simply members of the community, working in the open with other members.
What that means in practice is that discussions happen on the mailing
list and code is available in a public repository. Would you consider
changing your mode of working to being more open?

[Sam Geeraerts]

Op Fri, 14 Oct 2016 10:07:59 +0100

Post by Bob Ham

Post by Bob Ham
- What are the decisions that need to be made for the planning of
gNewSense 5?

There's no response to this question. Should we assume that there are
no pending decisions which need to be made?

Assuming that Builder will work, nothing substantial. Think of a name,
maybe come up with some new artwork, ...

Post by Bob Ham
if there is going to be effective team work, there needs to be much
better communication.

Agreed.

Post by Bob Ham
I'd like to ask: is there a particular reason for holding
these discussions in private?

I try to make the gNewSense community friendly and welcoming. But you
don't have to look far to find examples that show that the free
software community can also be a hostile environment. This can make
people who are otherwise enthusiastic hesitant to take their first
steps in public. Discussion in private can help build their confidence.

There can also be other reasons to initiate discussion in private, e.g.
privacy issues that are useful for a mentor to know, but don't need to
be shared with everyone.

Post by Bob Ham
I'm concerned by this mode of working. It seems that there is (Sam
+Eddi) doing work in private who will then seek feedback from some
separate entity called "the community". What I would expect, and what
is the norm in modern bazaar-style development is that Sam and Eddi
are simply members of the community, working in the open with other
members. What that means in practice is that discussions happen on
the mailing list and code is available in a public repository. Would
you consider changing your mode of working to being more open?

Of course. It's not my intention to do all development in back rooms
and only make it public when it's finished. I'll come back on this in
the next few days.

[Sam Geeraerts]

I've updated the project page for gNS 5 [1]. I'm replicating the plan
of approach below (wiki syntax). Those who are interested in helping
should sign up as a gNewSense member on Savannah to get VCS write
access.

We will want to have Ucclia and Ucclia+1 in a single repo, so that
users won't have to change URLs in sources.list. So checking how we can
merge multiple repos is an important factor in the success of the
Builder approach.

== Plan of approach ==
1. Set up VCS repo.
1. Push Builder code back to Savannah.
1. Announce on gnewsense-dev and assemble interested contributors.
1. Give contributors write access to VCS repo.
1. Create branch for every contributor. Sam acts as gatekeeper to
master.
1. Builder acts on a single distribution. Find a way to merge multiple
distributions into 1.
1. Update Builder to Debian.
1. Strip down Builder to its core (remove gen scripts etc.)
1. Update Builder for Debian without -security.
1. Update Builder for Debian with security.
1. Write package modifiers.
1. Use Builder's packages/ mechanism (so we can have packages-ucclia
style development) or use gen-scripts?
1. Go live with new repo.
1. Make Builder deb package.


[1] http://www.gnewsense.org/DevelopmentProjects/Gnewsense5

[l***@riseup.net]

Post by Sam Geeraerts
We will want to have Ucclia and Ucclia+1 in a single repo, so that
users won't have to change URLs in sources.list. So checking how we can
merge multiple repos is an important factor in the success of the
Builder approach.

I will test pulling the debian wheezy mirror into the debian jessie
mirror using following flags[1]
to see if the mirror is not purged of any jessie packages/files:

--ignore=regex
Never delete any files whose filenames match the regex. May be used
multiple times.

--nocleanup
Do not clean up the local mirror.


eddi

[1] https://linux.die.net/man/1/debmirror

[dww]

Hello Sam,

I look forward to participating again in helping with gNewSense5.
I imagine the kind of freedom bugs I worked in 4 are going to be a
priority.

Will gNewSense5 be based on Debian Jessie version 8.6?

On my development machine does it makes sense for me to start by
installing Jessie 8.6?

Regarding learning how to use the Builder approach is their some primer
or other learning material from the Debian or other sites?

Regards,
Dennis

Post by Sam Geeraerts
I've updated the project page for gNS 5 [1]. I'm replicating the plan
of approach below (wiki syntax). Those who are interested in helping
should sign up as a gNewSense member on Savannah to get VCS write
access.
We will want to have Ucclia and Ucclia+1 in a single repo, so that
users won't have to change URLs in sources.list. So checking how we can
merge multiple repos is an important factor in the success of the
Builder approach.
== Plan of approach ==
1. Set up VCS repo.
1. Push Builder code back to Savannah.
1. Announce on gnewsense-dev and assemble interested contributors.
1. Give contributors write access to VCS repo.
1. Create branch for every contributor. Sam acts as gatekeeper to
master.
1. Builder acts on a single distribution. Find a way to merge multiple
distributions into 1.
1. Update Builder to Debian.
1. Strip down Builder to its core (remove gen scripts etc.)
1. Update Builder for Debian without -security.
1. Update Builder for Debian with security.
1. Write package modifiers.
1. Use Builder's packages/ mechanism (so we can have packages-ucclia
style development) or use gen-scripts?
1. Go live with new repo.
1. Make Builder deb package.
[1] http://www.gnewsense.org/DevelopmentProjects/Gnewsense5
_______________________________________________
gNewSense-dev mailing list
https://lists.nongnu.org/mailman/listinfo/gnewsense-dev

[Sam Geeraerts]

Op Tue, 18 Oct 2016 19:38:37 -0400

Post by dww
I look forward to participating again in helping with gNewSense5.
I imagine the kind of freedom bugs I worked in 4 are going to be a
priority.

Great.

Post by dww
Will gNewSense5 be based on Debian Jessie version 8.6?

Yes.

Post by dww
On my development machine does it makes sense for me to start by
installing Jessie 8.6?

You can also work in gNewSense 4 by adding Jessie's deb-src links to
your sources.list.

Post by dww
Regarding learning how to use the Builder approach is their some
primer or other learning material from the Debian or other sites?

There's some documentation about Builder on the website [1]. Running
Builder needs quite a bit of disk space and takes some time. If you're
more interested in patching packages, then Builder's gen scripts (e.g.
gen-apt) are most interesting. They can be run by themselves as long as
you have the config file.

Builder also allows for Ucclia style patching, except it needs the full
package source and not just the debian folder (although it shouldn't be
hard to support that too). Both development approaches have their
strong and weak points. I'm not yet sure if we're going to use one or
the other or both.

Don't hesitate to ask questions. Builder can be daunting at first.

[1]
http://www.gnewsense.org/Builder/HowToCreateYourOwnGNULinuxDistribution

[dww]

Hello Sam,

If you don't mind, I have some basic questions about using Builder.

Post by Sam Geeraerts

Post by dww
Regarding learning how to use the Builder approach is their some
primer or other learning material from the Debian or other sites?

There's some documentation about Builder on the website [1].

I have been reading this page, buy didn't understand the overall
workflow.(forced to spend too much time in the Microsoft build world).

Would you please provide a basic description of the overall process in
terms of the items that are created on the local machine and the
relation to external sources of source code and packages?

Is the procedure creating a local repository of upstream source code and
build artifacts and gNewSense modified source code and build artifacts
starting with those from an upstream mirror defined in MIRROR?

Would you please explain what the different mirrors are (MIRRORLOCAL and
MIRROR)? In the text in Step 3 for MIRROR it refers to step 5, but
step 5 is about installing certain packages needed to run Builder
properly.

In step 4 I think debmirror is used to create a local mirror at location
in MIRRORLOCAL? Is this the mirror to which the build from the
repository generated and updated in steps 6 and 7 is pushed in step 9.

Or is MIRRORLOCAL the mirror that should be a location on a distribution
server?

Does ./do-update actually build binaries from code in the repository or
I should ask specifically what do debmirror and do-update accomplish?

(Note)In step 7 should the pushing of changes be done in step 9 not 7?)

I would appreciate any description of what the different steps do rather
than just how?

Post by Sam Geeraerts
Running Builder needs quite a bit of disk space and takes some time.

Could I run Builder from an external hard drive such as a Western
Digital WD Elements which would guarantee sufficient disk space?

I noticed that the in the config file for Builder from the gNewSense
Savannah repository that BASEDIR and MIRRORLOCAL are in srv in the file
system. Can this be on an external drive instead?

Post by Sam Geeraerts
If you're
more interested in patching packages, then Builder's gen scripts (e.g.
gen-apt) are most interesting. They can be run by themselves as long as
you have the config file.

So basically, patching packages involves creating gen script files with
basically sed scripts for applying changes to package source files?

What is the procedure for building and testing the modified packages,
that is, how are the gen scripts run by themselves along with the config
file? Is it just run as a bash script?

Post by Sam Geeraerts
Builder also allows for Ucclia style patching, except it needs the full
package source and not just the debian folder (although it shouldn't be
hard to support that too). Both development approaches have their
strong and weak points. I'm not yet sure if we're going to use one or
the other or both.
Don't hesitate to ask questions. Builder can be daunting at first.

Thank you.
Your help would be highly appreciated.

Dennis

Post by Sam Geeraerts
[1]
http://www.gnewsense.org/Builder/HowToCreateYourOwnGNULinuxDistribution
_______________________________________________
gNewSense-dev mailing list
https://lists.nongnu.org/mailman/listinfo/gnewsense-dev

[dww]

Hello Sam,

Regarding the instructions to only modify a package, are the
instructions in http://www.gnewsense.org/Builder/ModifyingANewPackage
applicable?

If yes, it appears that there is not an automated way to extract the
changes one made and generate the sed script for the gen script file?
I suppose though the manual addition to the gen script would probably
done once and rarely additional times?

Thanks,
Dennis

Post by dww
Hello Sam,
If you don't mind, I have some basic questions about using Builder.

Post by Sam Geeraerts

Post by dww
Regarding learning how to use the Builder approach is their some
primer or other learning material from the Debian or other sites?

There's some documentation about Builder on the website [1].

I have been reading this page, buy didn't understand the overall
workflow.(forced to spend too much time in the Microsoft build world).
Would you please provide a basic description of the overall process in
terms of the items that are created on the local machine and the
relation to external sources of source code and packages?
Is the procedure creating a local repository of upstream source code and
build artifacts and gNewSense modified source code and build artifacts
starting with those from an upstream mirror defined in MIRROR?
Would you please explain what the different mirrors are (MIRRORLOCAL and
MIRROR)? In the text in Step 3 for MIRROR it refers to step 5, but
step 5 is about installing certain packages needed to run Builder
properly.
In step 4 I think debmirror is used to create a local mirror at location
in MIRRORLOCAL? Is this the mirror to which the build from the
repository generated and updated in steps 6 and 7 is pushed in step 9.
Or is MIRRORLOCAL the mirror that should be a location on a distribution
server?
Does ./do-update actually build binaries from code in the repository or
I should ask specifically what do debmirror and do-update accomplish?
(Note)In step 7 should the pushing of changes be done in step 9 not 7?)
I would appreciate any description of what the different steps do rather
than just how?

Post by Sam Geeraerts
Running Builder needs quite a bit of disk space and takes some time.

Could I run Builder from an external hard drive such as a Western
Digital WD Elements which would guarantee sufficient disk space?
I noticed that the in the config file for Builder from the gNewSense
Savannah repository that BASEDIR and MIRRORLOCAL are in srv in the file
system. Can this be on an external drive instead?

Post by Sam Geeraerts
If you're
more interested in patching packages, then Builder's gen scripts (e.g.
gen-apt) are most interesting. They can be run by themselves as long as
you have the config file.

So basically, patching packages involves creating gen script files with
basically sed scripts for applying changes to package source files?
What is the procedure for building and testing the modified packages,
that is, how are the gen scripts run by themselves along with the config
file? Is it just run as a bash script?

Post by Sam Geeraerts
Builder also allows for Ucclia style patching, except it needs the full
package source and not just the debian folder (although it shouldn't be
hard to support that too). Both development approaches have their
strong and weak points. I'm not yet sure if we're going to use one or
the other or both.
Don't hesitate to ask questions. Builder can be daunting at first.

Thank you.
Your help would be highly appreciated.
Dennis

Post by Sam Geeraerts
[1]
http://www.gnewsense.org/Builder/HowToCreateYourOwnGNULinuxDistribution
_______________________________________________
gNewSense-dev mailing list
https://lists.nongnu.org/mailman/listinfo/gnewsense-dev

[Sam Geeraerts]

Op Sat, 24 Dec 2016 10:57:04 -0500

Post by dww
Regarding the instructions to only modify a package, are the
instructions in http://www.gnewsense.org/Builder/ModifyingANewPackage
applicable?

Yes.

Post by dww
If yes, it appears that there is not an automated way to extract the
changes one made and generate the sed script for the gen script file?

The sed script needs to be manually written. If you like using patches
instead you can modify the package the Ucclia way first and then put
the patch in a here file in the gen script + some code to make quilt
aware of it.

Post by dww
I suppose though the manual addition to the gen script would probably
done once and rarely additional times?

Yes. We only change a gen script when upstream modifies something that
breaks it or when we have new insights that require further
modifications to a package.

[dww]

Post by dww
Regarding the instructions to only modify a package, are the
instructions in http://www.gnewsense.org/Builder/ModifyingANewPackage
applicable?

Here is the debian/rules file content from lensfun-0.2.4:

[dww]

This email got sent accidently here is the complete email

Post by dww
Regarding the instructions to only modify a package, are the
instructions in http://www.gnewsense.org/Builder/ModifyingANewPackage
applicable?

In the instructions it says to execute:

fakeroot debian/rules binary

However some rules files do not have a binary target such as below

Here is the debian/rules file content from lensfun-0.2.4:

#!/usr/bin/make -f

include /usr/share/cdbs/1/rules/debhelper.mk
include /usr/share/cdbs/1/class/makefile.mk
include /usr/share/cdbs/1/rules/patchsys-quilt.mk

configure/liblensfun-dev::
./configure --prefix=/usr --libdir=/usr/lib

EB_MAKE_BUILD_TARGET := all
DEB_MAKE_INSTALL_TARGET := install DESTDIR=$(CURDIR)/debian/tmp/
DEB_MAKE_CHECK_TARGET := tests
DEB_DH_MAKESHLIBS_ARGS_ALL := -- -v0.2.3


In this case what happens are all the lines executed?

Also the instructions indicate that when the updated package is
completed send it to the "List". Then later someone will do the
do-update and push. Will the same procedure be followed?

Alternatively, the one updating the package could do the do-update
locally and push to the public repository?

Thanks,
Dennis

Post by dww

Post by dww
Regarding the instructions to only modify a package, are the
instructions in http://www.gnewsense.org/Builder/ModifyingANewPackage
applicable?

_______________________________________________
gNewSense-dev mailing list
https://lists.nongnu.org/mailman/listinfo/gnewsense-dev

[Sam Geeraerts]

Op Mon, 26 Dec 2016 10:39:07 -0500

Post by dww
fakeroot debian/rules binary
However some rules files do not have a binary target such as below

There's debhelper magic involved there. debian/rules includes
debhelper.mk, which in turn includes buildcore.mk and that defines a
default binary target.

Post by dww
Also the instructions indicate that when the updated package is
completed send it to the "List". Then later someone will do the
do-update and push. Will the same procedure be followed?

You will most likely make your changes in your own bzr branch. If you
let this mailing list know about a change, then we can review it and I
will merge it into the main branch from which Builder will update
itself.

Post by dww
Alternatively, the one updating the package could do the do-update
locally and push to the public repository?

To make sure that the public repository remains in a consistent state
it must be updated by a single Builder instance. So the best workflow
is to merge code changes to a master bzr branch and run the official
Builder from that code.

[Sam Geeraerts]

Op Fri, 23 Dec 2016 23:26:48 -0500

Post by dww
Is the procedure creating a local repository of upstream source code
and build artifacts and gNewSense modified source code and build
artifacts starting with those from an upstream mirror defined in
MIRROR?

It creates a local full (source + binaries) repository of upstream in
MIRROR. The gNewSense repo is initially a copy of that. As you add gen
scripts, you get gNewSense specific packages in the gNewSense repo.

Post by dww
Would you please explain what the different mirrors are (MIRRORLOCAL
and MIRROR)? In the text in Step 3 for MIRROR it refers to step 5,
but step 5 is about installing certain packages needed to run Builder
properly.

MIRRORLOCAL is the path to the upstream mirror on your hard drive.
MIRROR is the same repo, but then as an HTTP path. Step 3 should refer
to step 4. Fixed now.

Post by dww
In step 4 I think debmirror is used to create a local mirror at
location in MIRRORLOCAL? Is this the mirror to which the build from
the repository generated and updated in steps 6 and 7 is pushed in
step 9.

MIRRORLOCAL remains unchanged, because that's upstream. The result of
the update goes in REPODST.

Post by dww
Or is MIRRORLOCAL the mirror that should be a location on a
distribution server?

What do you mean by "distribution server"?

Post by dww
Does ./do-update actually build binaries from code in the repository
or I should ask specifically what do debmirror and do-update
accomplish?

Debmirror syncs with upstream, do-update checks if a package needs
updating and runs the gen script if it does.

Post by dww
(Note)In step 7 should the pushing of changes be done in step 9 not 7?)

Indeed. Fixed now.

Post by dww
I would appreciate any description of what the different steps do
rather than just how?

1. Generate a key to sign your gNewSense repo.
2. Get the Builder source code so that you can run it.
3. Configure Builder for your needs.
4. Get the upstream repo.
5. Install Builder's runtime dependencies.
6. Set up your gNewSense repo, i.e. write reprepro's configuration.
7. Fill your gNewSense repo and modify packages that require it.
8. Generate a live CD. You run this only with a new release.
9. Make your repo public (all the previous steps happened in folders
that the outside world doesn't see).

Post by dww
Could I run Builder from an external hard drive such as a Western
Digital WD Elements which would guarantee sufficient disk space?

Sure. It might be a bit slower, but technically there's no problem.

Post by dww
I noticed that the in the config file for Builder from the gNewSense
Savannah repository that BASEDIR and MIRRORLOCAL are in srv in the
file system. Can this be on an external drive instead?

It can be anywhere you like.

Post by dww
So basically, patching packages involves creating gen script files
with basically sed scripts for applying changes to package source
files?

A gen script does the following:

1. Unpack the source package.
2. Modify the source code.
3. Repack the source package.
4. Build the binary packages.

It's the same as what you do when you modify a package by hand. The
second step does what you would do with an editor. Sed is a convenient
way to change text, but you could also use another method. You could
also create patches like we do for Ucclia and put the commands to do
that in the gen script. E.g.:

cat <<HERE >/tmp/my_patch
--- a/foo/bar
+++ b/foo/bar
@@ -1,2 +1,2 @@
-This is old.
+This is new.
The end.
HERE
quilt import /tmp/my_patch

The advantage of sed is that you don't have to change the scripts if
line numbers change.

Post by dww
What is the procedure for building and testing the modified packages,
that is, how are the gen scripts run by themselves along with the
config file? Is it just run as a bash script?

It's just a bash script that you can run without running all of Builder.

[dww]

Post by Sam Geeraerts
MIRRORLOCAL is the path to the upstream mirror on your hard drive.
MIRROR is the same repo, but then as an HTTP path.

Why then is MIRROR needed? If the repository is created and updated on
the same machine as MIRRORLOCAL and then pushed to a public mirror
cannot the script that does the update use MIRRORLOCAL directly as the
path to the source and binaries? How is the http link defined in MIRROR
linked to the file system location in MIRRORLOCAL? In general where in
the process is MIRRORLOCAL or MIRROR used? What is the useful purpose of
MIRROR?

Post by Sam Geeraerts
9. Make your repo public (all the previous steps happened in folders
that the outside world doesn't see).

Step 9 refers to publishing or pushing the repo to "your mirror".
Looking at push-repo this appears to be the location configured in
RSYNC_DEST. In the case of Dunsink where do you think RSYNC_DEST will
point to? Will it be some public location at Savannah or a gNewSense
server? Who uses this final mirror location?

So if I understand the process, MIRRORLOCAL, MIRROR are created on the
machine of the person doing the update. Creates and updates the
repository on their machine then pushes the result to the location
defined in RSYNC_DEST. Is this correct? This is why I do not
understand the purpose of MIRROR.


Where does pbuilder come into play in this process or is that a Ubuntu
wrapper around the builder process?

[Sam Geeraerts]

Op Mon, 26 Dec 2016 10:23:51 -0500

Post by dww
Why then is MIRROR needed? If the repository is created and updated on
the same machine as MIRRORLOCAL and then pushed to a public mirror
cannot the script that does the update use MIRRORLOCAL directly as the
path to the source and binaries? How is the http link defined in
MIRROR linked to the file system location in MIRRORLOCAL? In general
where in the process is MIRRORLOCAL or MIRROR used? What is the
useful purpose of MIRROR?

MIRROR is used in gen-repo for reprepro's configuration. It needs a URL
that apt-get can access (the URL you use in sources.list). I believe
that can also be a file:// URL now, but I'm not sure reprepro could
handle that back in the day. In that case MIRROR could be just
"file://" + MIRRORLOCAL.

In case of an http:// URL the link is defined in the web server that
serves that repo.

Post by dww
Step 9 refers to publishing or pushing the repo to "your mirror".
Looking at push-repo this appears to be the location configured in
RSYNC_DEST. In the case of Dunsink where do you think RSYNC_DEST will
point to? Will it be some public location at Savannah or a gNewSense
server? Who uses this final mirror location?

Builder is designed so that you can have separate machines for
compiling and for serving your repo. Building happens on a not so
public server, the final location is archive.gnewsense.org a.k.a. the
master repo server.

Post by dww
So if I understand the process, MIRRORLOCAL, MIRROR are created on the
machine of the person doing the update. Creates and updates the
repository on their machine then pushes the result to the location
defined in RSYNC_DEST. Is this correct? This is why I do not
understand the purpose of MIRROR.

Builder is initially set up on a build server and runs in a cron job
after that. It automatically pushes the result to the master repo
server. The only human interaction required is writing gen-scripts.
Note that Builder automatically syncs its source code with the main bzr
branch (see update-builder).

Post by dww
Where does pbuilder come into play in this process or is that a Ubuntu
wrapper around the builder process?

Builder doesn't use pbuilder, although I think it would be a good
addition. Builder uses chroots directly, while pbuilder is a wrapper
around chroot. Switching to pbuilder would probably make Builder run
(even) longer, but I think it fits the build process of deb packages
better.

[dww]

Hello Sam,

I was thinking of trying the builder process out on my dev machine. In
general I think builder is a better process than the old Ucclia update
process.

I noticed in Savannah gNewSense repository there is builder,
builder_lordeddi, and build_samgee.

Are any of these useful to try out? I think builder goes back to deltah,
the last gNewSense based on Ubuntu.

Is an update for Dunsink in progress?

Thanks,
Dennis

[Sam Geeraerts]

Op Mon, 26 Dec 2016 10:52:46 -0500

Post by dww
Are any of these useful to try out? I think builder goes back to
deltah, the last gNewSense based on Ubuntu.
Is an update for Dunsink in progress?

builder_lordeddi is your best bet.

[dww]

Post by Sam Geeraerts
Op Mon, 26 Dec 2016 10:52:46 -0500

Post by dww
Are any of these useful to try out? I think builder goes back to
deltah, the last gNewSense based on Ubuntu.
Is an update for Dunsink in progress?

builder_lordeddi is your best bet.

In step 3 of the builder process it states:

apt-get install mr
cd packages
mkdir deltah{,-security,-updates,-backports}
mr -c gnewsense.mrconfig checkout


The directory names include "deltah" as well as the links in
gnewsense.mrconfig.

For the short term can I leave the directory names with "deltah"? Will
the other steps work OK for now?
I would think the links in gnewsense.mrconfig will eventually need to
contain "dunksink" instead of "deltah" as well as those packages need to
be updated to build and work with "dunksink"?

Thanks,
Dennis

[Sam Geeraerts]

Op Fri, 30 Dec 2016 08:53:04 -0500

Post by dww
apt-get install mr
cd packages
mkdir deltah{,-security,-updates,-backports}
mr -c gnewsense.mrconfig checkout
The directory names include "deltah" as well as the links in
gnewsense.mrconfig.
For the short term can I leave the directory names with "deltah"?
Will the other steps work OK for now?

Eddi already changed the config file to dunsink. The do-update script
looks for packages/$RELEASE*, so you should create dunsink folders. You
should take the name (and upstream) change into account for the other
steps as well.

[Worthem, Dennis]

Post by Sam Geeraerts
Eddi already changed the config file to dunsink. The do-update script
looks for packages/$RELEASE*, so you should create dunsink folders. You
should take the name (and upstream) change into account for the other
steps as well.

The links in gnewsense.mrconfig have not been changed. Will builder
work with the old deltah versions of this packages even though the
folders have been changed to dunsink?

[Worthem, Dennis]

Post by Sam Geeraerts
Eddi already changed the config file to dunsink. The do-update script
looks for packages/$RELEASE*, so you should create dunsink folders. You
should take the name (and upstream) change into account for the other
steps as well.

The links in gnewsense.mrconfig have not been changed. Will builder work with the old deltah versions of this packages even though the folders have been changed to dunsink?

I changed the section headers in gnewsense.mrconfig but left the links
to deltah as there is none now for dunsink.

For example:

[dunsink-security/openoffice.org]
chain = false
checkout = bzr branch
bzr://bzr.gnewsense.org/packages-deltah-security/openoffice.org
openoffice.org

Will this work for updating the repository?

[Sam Geeraerts]

Op Fri, 30 Dec 2016 18:00:50 -0500

Post by Worthem, Dennis
The links in gnewsense.mrconfig have not been changed. Will builder
work with the old deltah versions of this packages even though the
folders have been changed to dunsink?

Could be, but I haven't tested that because they will need to be
updated anyway. E.g. the artwork has changed and OpenOffice.org is now
obsolete. The important thing is that the principle of this mechanism
works so that we can add our own packages to the repo.

[Worthem, Dennis]

Hello Sam,

I have been working on setting up the http (Apache) setting for the
builder environment.

For example, for REPOAPT=http://127.0.0.1/$DISTRONAME_L or
localhost/$DISTRONAME_L

where DISTRONAME_L is gnewsense.

I created a symbolic link from /var/www/html/ as gnewsense to
/media/dennis/Elements/builder/srv/gnewsense.

where /var/www/html contains index.html which is the Apache 2 Debian
default page. Also Elements is the volume name of my external drive.

In the browser, localhost, brings up the default page. When I enter
localhost/gnewsense I get the 403 Forbidden error and in the error.log

I have the error:

Symbolic link not allowed or link target not accessible:
/var/www/html/gnewsense

It is not clear to me how to fix this after some internet searches.

Does the apache user (www-data) have to be given read and or execute
privileges to /media/dennis/Elements/builder/srv/gnewsense?

Thanks much,

Dennis

[Worthem, Dennis]

Post by dww
Hello Sam,
I have been working on setting up the http (Apache) setting for the builder environment.
For example, for REPOAPT=http://127.0.0.1/$DISTRONAME_L or localhost/$DISTRONAME_L
where DISTRONAME_L is gnewsense.
I created a symbolic link from /var/www/html/ as gnewsense to /media/dennis/Elements/builder/srv/gnewsense.
where /var/www/html contains index.html which is the Apache 2 Debian default page. Also Elements is the volume name of my external drive.
In the browser, localhost, brings up the default page. When I enter localhost/gnewsense I get the 403 Forbidden error and in the error.log
Symbolic link not allowed or link target not accessible: /var/www/html/gnewsense
It is not clear to me how to fix this after some internet searches.
Does the apache user (www-data) have to be given read and or execute privileges to /media/dennis/Elements/builder/srv/gnewsense?
Thanks much,
Dennis

I checked with the line "sudo -u www-data ls /var/www/html/gnewsense'

and said permission was denied.

I tried to set read and execute rights for www-data to symlinked folder
and could not get it to work

[Sam Geeraerts]

Op Sun, 01 Jan 2017 17:27:36 -0500

Post by dww
Hello Sam,
I have been working on setting up the http (Apache) setting for the
builder environment.
For example, for REPOAPT=http://127.0.0.1/$DISTRONAME_L or
localhost/$DISTRONAME_L
where DISTRONAME_L is gnewsense.
I created a symbolic link from /var/www/html/ as gnewsense to
/media/dennis/Elements/builder/srv/gnewsense.
where /var/www/html contains index.html which is the Apache 2 Debian
default page. Also Elements is the volume name of my external drive.
In the browser, localhost, brings up the default page. When I enter
localhost/gnewsense I get the 403 Forbidden error and in the
error.log
/var/www/html/gnewsense
It is not clear to me how to fix this after some internet searches.
Does the apache user (www-data) have to be given read and or execute
privileges to /media/dennis/Elements/builder/srv/gnewsense?

www-data needs read rights on /var/www/html/gnewsense
and /media/dennis/Elements/builder/srv/gnewsense and below. It also
needs execute rights on /media/dennis/Elements/builder/srv/gnewsense
and below. You may also need to set +FollowSymLinks in Apache's
configuration.

[Worthem, Dennis]

Post by Sam Geeraerts
www-data needs read rights on /var/www/html/gnewsense
and /media/dennis/Elements/builder/srv/gnewsense and below. It also
needs execute rights on /media/dennis/Elements/builder/srv/gnewsense
and below. You may also need to set +FollowSymLinks in Apache's
configuration.

For example I tried the following:

sudo usermod -a -G www-data dennis

sudo chgrp www-data /media/dennis/Elements/builder/srv/gnewsense

sudo chmod g+rwxs /media/dennis/Elements/builder/srv/gnewsense

then

ls -la for /media/dennis/Elements/builder/srv/gnewsense

with the results:

total 12
drwx------ 1 dennis dennis 0 Dec 31 18:37 .
drwx------ 1 dennis dennis 0 Dec 31 18:20 ..
-rw------- 1 dennis dennis 10701 Dec 31 17:44 index.html

did the same for /var/www/html/gnewsense

with the same results for ls -la

Is there anything else I need to do?

[l***@riseup.net]

Hi Dennis,

Post by Worthem, Dennis
ls -la for /media/dennis/Elements/builder/srv/gnewsense
total 12
drwx------ 1 dennis dennis 0 Dec 31 18:37 .
drwx------ 1 dennis dennis 0 Dec 31 18:20 ..
-rw------- 1 dennis dennis 10701 Dec 31 17:44 index.html
did the same for /var/www/html/gnewsense
with the same results for ls -la
Is there anything else I need to do?

you still get the 403 ?
I also use an external HD, but mount it to /srv (which i created)
using nginx to serve the files i had a similar problem.

i had to make the /srv folder owned by www-data:www-data or i get a 403
(user and group)
I think that if nginx encounters a folder not owned by it's user on the
path to the files you want to serve it throws a 403, i assume apache has
a similar policy

not sure if this helps

Post by Worthem, Dennis
_______________________________________________
gNewSense-dev mailing list
https://lists.nongnu.org/mailman/listinfo/gnewsense-dev

[Sam Geeraerts]

Op Sun, 08 Jan 2017 13:24:02 -0500

Post by Worthem, Dennis
sudo chgrp www-data /media/dennis/Elements/builder/srv/gnewsense
sudo chmod g+rwxs /media/dennis/Elements/builder/srv/gnewsense

You should run these recursively (-R) to make everything inside the
folder readable for www-data.

Post by Worthem, Dennis
ls -la for /media/dennis/Elements/builder/srv/gnewsense
total 12
drwx------ 1 dennis dennis 0 Dec 31 18:37 .
drwx------ 1 dennis dennis 0 Dec 31 18:20 ..
-rw------- 1 dennis dennis 10701 Dec 31 17:44 index.html

As you can see here, the file is still owned by group dennis. The
parent directory is probably owned by group www-data.

[Bob Ham]

Post by Sam Geeraerts
1. Set up VCS repo.
1. Push Builder code back to Savannah.
1. Announce on gnewsense-dev and assemble interested contributors.
1. Give contributors write access to VCS repo.
1. Create branch for every contributor. Sam acts as gatekeeper to
master.

Will you be doing this, Sam? Again, I don't want to make assumptions.
Should I/we be waiting for an email from you to say that the repo is
good to go?

Thanks,

Bob

[Sam Geeraerts]

Op Tue, 25 Oct 2016 09:23:13 +0100

Post by Bob Ham

Post by Sam Geeraerts
1. Create branch for every contributor. Sam acts as gatekeeper to
master.

Will you be doing this, Sam? Again, I don't want to make assumptions.
Should I/we be waiting for an email from you to say that the repo is
good to go?

Please go right ahead and create your own branch. I prefer that you
push it as builder_<username> to Savannah. If you made a change that
you would like to see merged into the master branch, please say so on
this mailing list.

[Bob Ham]

Post by Sam Geeraerts
Op Tue, 25 Oct 2016 09:23:13 +0100

Post by Bob Ham

Post by Sam Geeraerts
1. Create branch for every contributor. Sam acts as gatekeeper to
master.

Will you be doing this, Sam? Again, I don't want to make assumptions.
Should I/we be waiting for an email from you to say that the repo is
good to go?

Please go right ahead and create your own branch. I prefer that you
push it as builder_<username> to Savannah. If you made a change that
you would like to see merged into the master branch, please say so on
this mailing list.

I'm a little confused. The first item under "Set up VCS repo" in the
plan is "Push Builder code back to Savannah." Have you done that? I
haven't been watching the repository. Looking now at Savannah,

http://bzr.savannah.gnu.org/lh/gnewsense/

there are two directories in the repo with "builder" in the name but
neither seem to have been updated since 2013.

There are two possible interpretations of the situation that I can see:
firstly, there were changes from 2013 which you've recently pushed back
to the repository where we can see the date of the change but not the
date of the push. Or secondly, that there are further changes to the
repository which are yet to be pushed back.

If it's the second case, I'd prefer to wait until the further changes
have been pushed so that I can branch from the latest code.

Could you clarify what the situation is?

Thanks,

Bob

[Sam Geeraerts]

Op Fri, 28 Oct 2016 10:46:19 +0100

Post by Bob Ham
I'm a little confused. The first item under "Set up VCS repo" in the
plan is "Push Builder code back to Savannah." Have you done that?

Yes. For technical reasons the Builder code lived on bzr.gnewsense.org
for a while. It's no longer necessary to keep it there and maintain our
own Bazaar+Loggerhead instance, so let's now just use Savannah's
infrastructure again. The version on Savannah now is the same as the
last version on bzr.gnewsense.org.

[Bob Ham]

Post by Sam Geeraerts
Op Fri, 28 Oct 2016 10:46:19 +0100

Post by Bob Ham
I'm a little confused. The first item under "Set up VCS repo" in the
plan is "Push Builder code back to Savannah." Have you done that?

Yes. For technical reasons the Builder code lived on bzr.gnewsense.org
for a while. It's no longer necessary to keep it there and maintain our
own Bazaar+Loggerhead instance, so let's now just use Savannah's
infrastructure again. The version on Savannah now is the same as the
last version on bzr.gnewsense.org.

Excellent, thanks :-)