• PI SDcard images

    From marty@3:770/3 to All on Saturday, June 11, 2022 13:24:03
    Hi folks,

    I have three PIs running with identical SanDisk Ultra 32GB SD-cards but
    each containing different OS versions and data.

    When I dd them to a backup image on my Linux desktop:

    sudo dd if=/dev/sde of=pibed-backup.bin bs=512 status=progress
    32008593920 bytes (32 GB, 30 GiB) copied, 1728 s, 18.5 MB/s
    62521344+0 records in
    62521344+0 records out
    32010928128 bytes (32 GB, 30 GiB) copied, 1728.62 s, 18.5 MB/s

    I get 3 different size images for the three SD-cards:

    marty@light:~/Desktop/downloads$ ls -l pi*
    -rw-r--r-- 1 root root 31981568000 Jun 5 17:58 pi-backup.bin
    -rw-r--r-- 1 root root 32010928128 Jun 10 17:14 pibed-backup.bin
    -rw-r--r-- 1 root root 31914983424 May 21 05:55 pihole-backup.bin

    Whats going on here. I expected the same size image for all three cards.
    Are SD-cards all slightly different sizes or is dd doing something funny dependent on the data on the cards?

    Cheers,

    --
    Marty

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From R.Wieser@3:770/3 to All on Saturday, June 11, 2022 07:59:44
    marty,

    Whats going on here. I expected the same size image for all three cards.
    Are SD-cards all slightly different sizes or is dd doing something funny dependent on the data on the cards?

    I had the same problem when I thought I could just duplicate one micro-sd
    card to another : It would not fit (a difference of 70 MB).

    The reason is most likely the number of sectors that are reserved by the
    cards internal controller (you can't see or touch them), so it can swap in
    new sectors for ones that get worn out.

    Regards,
    Rudy Wieser

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From marty@3:770/3 to R.Wieser on Saturday, June 11, 2022 16:35:26
    On 11/6/22 15:59, R.Wieser wrote:
    marty,

    Whats going on here. I expected the same size image for all three cards.
    Are SD-cards all slightly different sizes or is dd doing something funny
    dependent on the data on the cards?

    I had the same problem when I thought I could just duplicate one micro-sd card to another : It would not fit (a difference of 70 MB).

    The reason is most likely the number of sectors that are reserved by the cards internal controller (you can't see or touch them), so it can swap in new sectors for ones that get worn out.

    Regards,
    Rudy Wieser



    That makes sense.

    I have ordered a 64GB card in case I ever need to make a new card from
    the 32GB backup image. Not sure what else to do, it is a hassle.

    Cheers,

    --
    Marty

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From R.Wieser@3:770/3 to All on Saturday, June 11, 2022 09:04:47
    marty,

    I have ordered a 64GB card in case I ever need to make a new card from the 32GB backup image. Not sure what else to do, it is a hassle.

    As far as I know DD can be told to only copy upto a certain size.

    Though the problems are than to :

    1) find some way to shrink the (last) partition (of a new installation) so
    at the end there is some unused space (200 MB ?)

    2) find out how to (automatically, scripted) obtain the "certain size" (mentioned above).

    I think I remember having seen a few posts about it, but can't remember what the outcome was.

    Regards,
    Rudy Wieser

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Ahem A Rivet's Shot@3:770/3 to marty on Saturday, June 11, 2022 08:03:36
    On Sat, 11 Jun 2022 13:24:03 +1000
    marty <marty@invalid.net> wrote:

    Whats going on here. I expected the same size image for all three cards.
    Are SD-cards all slightly different sizes or is dd doing something funny dependent on the data on the cards?

    The former, the same applies to hard disks too and sometimes causes problems when replacing drives in a RAID setup.

    --
    Steve O'Hara-Smith
    Odds and Ends at http://www.sohara.org/

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Ahem A Rivet's Shot@3:770/3 to R.Wieser on Saturday, June 11, 2022 09:07:42
    On Sat, 11 Jun 2022 09:04:47 +0200
    "R.Wieser" <address@not.available> wrote:

    1) find some way to shrink the (last) partition (of a new installation)
    so at the end there is some unused space (200 MB ?)

    Not sure about SD cards but when I hit this with 2TB disc drives
    the spread in sizes was only a few megabytes.

    2) find out how to (automatically, scripted) obtain the "certain size" (mentioned above).

    I think I remember having seen a few posts about it, but can't remember
    what the outcome was.

    Fix the tool that expands the partition in the first place to round
    the size down to a multiple of 16Mb instead of using it all.

    --
    Steve O'Hara-Smith
    Odds and Ends at http://www.sohara.org/

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From A. Dumas@3:770/3 to marty on Saturday, June 11, 2022 10:09:30
    marty <marty@invalid.net> wrote:
    I have ordered a 64GB card in case I ever need to make a new card from
    the 32GB backup image. Not sure what else to do, it is a hassle.

    You don't have to have exactly matching sd cards or larger ones like 64 GB.
    See the top answer here https://askubuntu.com/questions/1174487/re-size-the-img-for-smaller-sd-card-how-to-shrink-a-bootable-sd-card-image
    which was the first hit for "linux shrink image file".

    It is indeed a hassle, unless you automate the process.

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From R.Wieser@3:770/3 to All on Saturday, June 11, 2022 13:31:05
    Ahem,

    Not sure about SD cards but when I hit this with 2TB disc
    drives the spread in sizes was only a few megabytes.

    I mentioned a difference of 70 MB. The OP supplied sizes with a difference
    of almost 96 MB.

    Fix the tool that expands the partition in the first place to
    round the size down to a multiple of 16Mb instead of using it all.

    I think you mean 'toolS', with an "s" on the end. Though in most cases
    there is no need for that. And why throw away storage space ?

    Just here, in the case of the RPi, it would have been nice to have a bit of 'slack space'.

    Regards,
    Rudy Wieser

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Ahem A Rivet's Shot@3:770/3 to R.Wieser on Saturday, June 11, 2022 13:33:52
    On Sat, 11 Jun 2022 13:31:05 +0200
    "R.Wieser" <address@not.available> wrote:

    Ahem,

    Not sure about SD cards but when I hit this with 2TB disc
    drives the spread in sizes was only a few megabytes.

    I mentioned a difference of 70 MB. The OP supplied sizes with a
    difference of almost 96 MB.

    OK - that makes it harder and more wasteful to find a size that
    fits all.

    I think you mean 'toolS', with an "s" on the end. Though in most cases
    there is no need for that. And why throw away storage space ?

    Yeah OK toolS - and because it's a small price to pay for
    convenience (even 256 megs wouldn't be much in 64 gigs or more), storage
    space is dirt cheap these days - hassle is expensive and irritating.

    --
    Steve O'Hara-Smith
    Odds and Ends at http://www.sohara.org/

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From R.Wieser@3:770/3 to All on Saturday, June 11, 2022 15:18:37
    Ahem,

    I mentioned a difference of 70 MB. The OP supplied sizes with
    a difference of almost 96 MB.

    OK - that makes it harder and more wasteful to find a size that
    fits all.

    I don't think there is any such size, as its fully upto the manufacturer of
    the micro-SD card to pick the number of replacement sectors - which get subtracted from the total size of the thumbdrive.

    Yeah OK toolS

    The problem is that all those tools get made by different people/companies.
    You would need to convince *all of them* to make such a change. There is little chance to that.

    and because it's a small price to pay for convenience

    As far as I can tell it would only be a possible convenience in regard to a
    RPi micro-SD card (and similar), and only to people who use the default
    instal. To all others it would be wasting space for no reason whatsoever.

    People who do a manual install (creating/defining their own partitions) its
    no problem to not to have the last partition extend upto the end of the
    storage space.

    Having said that and thinking back to a number of "noobs" installs I sure
    would have liked to have had a choice in it.

    Regards,
    Rudy Wieser

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Dennis Lee Bieber@3:770/3 to All on Saturday, June 11, 2022 11:32:25
    On Sat, 11 Jun 2022 13:24:03 +1000, marty <marty@invalid.net> declaimed the following:

    Hi folks,

    I have three PIs running with identical SanDisk Ultra 32GB SD-cards but
    each containing different OS versions and data.


    Whats going on here. I expected the same size image for all three cards.
    Are SD-cards all slightly different sizes or is dd doing something funny >dependent on the data on the cards?

    Unless those cards are all from the same production batch, it IS possible that they are slightly different internally. Controllers reserving different amounts of "allocation units" for wear leveling. Possibly some bad-block remapping even at the factory floor.

    With a good light, and a good magnifier, you might be able to read the production code off the card -- if the code differs, anything goes.

    There may also be an effect from how the FAT partition was sized vs the EXT# partition(s)


    --
    Wulfraed Dennis Lee Bieber AF6VN
    wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From marty@3:770/3 to A. Dumas on Sunday, June 12, 2022 12:38:06
    On 11/6/22 20:09, A. Dumas wrote:
    marty <marty@invalid.net> wrote:
    I have ordered a 64GB card in case I ever need to make a new card from
    the 32GB backup image. Not sure what else to do, it is a hassle.

    You don't have to have exactly matching sd cards or larger ones like 64 GB. See the top answer here https://askubuntu.com/questions/1174487/re-size-the-img-for-smaller-sd-card-how-to-shrink-a-bootable-sd-card-image
    which was the first hit for "linux shrink image file".

    It is indeed a hassle, unless you automate the process.

    I used the script pishrink.sh mentioned on that web page and it worked!
    gparted doesn't like the resulting SD card (error: can't have partition
    outside disk space) but it boots in the PI.

    Cheers,

    --
    Marty

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Knute Johnson@3:770/3 to marty on Sunday, June 12, 2022 09:36:55
    For the SD cards that have a Raspberry image on them you can use the SD
    Card Copier program. I just backed up my 240GB SSD drive to a 64GB uSD
    card. This won't work of course if you have more stuff on the card than
    will fit.

    On 6/10/22 22:24, marty wrote:
    Hi folks,

    I have three PIs running with identical SanDisk Ultra 32GB SD-cards but
    each containing different OS versions and data.

    When I dd them to a backup image on my Linux desktop:

    sudo dd if=/dev/sde of=pibed-backup.bin bs=512 status=progress
    32008593920 bytes (32 GB, 30 GiB) copied, 1728 s, 18.5 MB/s
    62521344+0 records in
    62521344+0 records out
    32010928128 bytes (32 GB, 30 GiB) copied, 1728.62 s, 18.5 MB/s

    I get 3 different size images for the three SD-cards:

    marty@light:~/Desktop/downloads$ ls -l pi*
    -rw-r--r-- 1 root root 31981568000 JunĀ  5 17:58 pi-backup.bin
    -rw-r--r-- 1 root root 32010928128 Jun 10 17:14 pibed-backup.bin
    -rw-r--r-- 1 root root 31914983424 May 21 05:55 pihole-backup.bin

    Whats going on here. I expected the same size image for all three cards.
    Are SD-cards all slightly different sizes or is dd doing something funny dependent on the data on the cards?

    Cheers,


    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From druck@3:770/3 to R.Wieser on Monday, June 13, 2022 21:09:29
    On 11/06/2022 08:04, R.Wieser wrote:
    marty,

    I have ordered a 64GB card in case I ever need to make a new card from the >> 32GB backup image. Not sure what else to do, it is a hassle.

    As far as I know DD can be told to only copy upto a certain size.

    Though the problems are than to :

    1) find some way to shrink the (last) partition (of a new installation) so
    at the end there is some unused space (200 MB ?)

    Use the gparted tool.

    2) find out how to (automatically, scripted) obtain the "certain size" (mentioned above).

    Just knock off 0.1GB from whatever size card you are using.

    ---druck

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From druck@3:770/3 to marty on Monday, June 13, 2022 21:17:12
    On 12/06/2022 03:38, marty wrote:
    On 11/6/22 20:09, A. Dumas wrote:
    marty <marty@invalid.net> wrote:
    I have ordered a 64GB card in case I ever need to make a new card from
    the 32GB backup image. Not sure what else to do, it is a hassle.

    You don't have to have exactly matching sd cards or larger ones like
    64 GB.
    See the top answer here
    https://askubuntu.com/questions/1174487/re-size-the-img-for-smaller-sd-card-how-to-shrink-a-bootable-sd-card-image

    which was the first hit for "linux shrink image file".

    It is indeed a hassle, unless you automate the process.

    I used the script pishrink.sh mentioned on that web page and it worked! gparted doesn't like the resulting SD card (error: can't have partition outside disk space) but it boots in the PI.

    Don't use that script it is wrong, and will cause problems with various
    disk tools of which gparted is only one.

    There is a way to fix a card which has been butchered in that way using
    the command line parted (not *g*parted) tool to delete the partition
    table entry and recreate it so it doesn't go over the end of the card.

    But its much easier to go back to the original larger card, and resize
    it properly with gparted before copying.

    ---druck

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Nikolaj Lazic@3:770/3 to All on Friday, June 17, 2022 20:02:10
    Dana Mon, 13 Jun 2022 21:09:29 +0100, druck <news@druck.org.uk> napis'o:
    On 11/06/2022 08:04, R.Wieser wrote:
    marty,

    I have ordered a 64GB card in case I ever need to make a new card from the >>> 32GB backup image. Not sure what else to do, it is a hassle.

    As far as I know DD can be told to only copy upto a certain size.

    Though the problems are than to :

    1) find some way to shrink the (last) partition (of a new installation) so >> at the end there is some unused space (200 MB ?)

    Use the gparted tool.

    And add partition at the end of the card. And make it start from the same sector on all cards. The end sector will be different, but it does not matter. And mark it as some other FS (for example ff).
    And than automatic expand will expand it to that place.


    2) find out how to (automatically, scripted) obtain the "certain size"
    (mentioned above).

    Just knock off 0.1GB from whatever size card you are using.

    ---druck


    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From scott@alfter.diespammersdie.us@3:770/3 to R.Wieser on Friday, June 17, 2022 20:46:51
    R.Wieser <address@not.available> wrote:
    As far as I know DD can be told to only copy upto a certain size.

    Though the problems are than to :

    1) find some way to shrink the (last) partition (of a new installation) so
    at the end there is some unused space (200 MB ?)

    gparted will do that for you. I have a project at work where I do just that
    to image it across multiple Raspberry Pis: use gparted to shrink the image
    size to the minimum needed to create the image, and to expand into the full space of a new SD card once the image is dd'd onto it.

    2) find out how to (automatically, scripted) obtain the "certain size" (mentioned above).

    I've not automated this, but once I have the partition size minimized, I use fdisk to get the last block of the last partition. Multiplying that by 2048 gets the image size in MB; pass that to dd as the "count" parameter, with "bs=1048576" to set the block size.

    --
    _/_
    / v \ Scott Alfter (remove the obvious to send mail)
    (IIGS( https://alfter.us/ Top-posting!
    \_^_/ >What's the most annoying thing on Usenet?

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From R.Wieser@3:770/3 to All on Saturday, June 18, 2022 10:01:55
    Scott,

    use gparted to shrink the image size to the minimum needed
    to create the image, and to expand into the full space of a
    new SD card once the image is dd'd onto it.

    gparted knows enough of the different filesystems to be able to do that ? Hmmmm... Maybe I should take a look at it.

    Although I like the idea of shrinking the backups to whats actually used, it also means I need to keep an un-shrunken (plain DD) backup with gparted on
    it to be able to do that ...

    Thanks for the info.

    Regards,
    Rudy Wieser

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Tauno Voipio@3:770/3 to scott@alfter.diespammersdie.us on Saturday, June 18, 2022 10:52:36
    On 17.6.22 23.46, scott@alfter.diespammersdie.us wrote:
    R.Wieser <address@not.available> wrote:
    As far as I know DD can be told to only copy upto a certain size.

    Though the problems are than to :

    1) find some way to shrink the (last) partition (of a new installation) so >> at the end there is some unused space (200 MB ?)

    gparted will do that for you. I have a project at work where I do just that to image it across multiple Raspberry Pis: use gparted to shrink the image size to the minimum needed to create the image, and to expand into the full space of a new SD card once the image is dd'd onto it.

    2) find out how to (automatically, scripted) obtain the "certain size"
    (mentioned above).

    I've not automated this, but once I have the partition size minimized, I use fdisk to get the last block of the last partition. Multiplying that by 2048 gets the image size in MB; pass that to dd as the "count" parameter, with "bs=1048576" to set the block size.


    No need for using fdisk there, gparted shows the numbers of blocks
    in partitions. Just check the last block of the last partition, add
    one and multiply with block size (usually 512).

    --

    -TV

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From scott@alfter.diespammersdie.us@3:770/3 to R.Wieser on Tuesday, June 21, 2022 06:14:12
    R.Wieser <address@not.available> wrote:
    gparted knows enough of the different filesystems to be able to do that ? Hmmmm... Maybe I should take a look at it.

    The only common filesystem I can think of that gparted might not handle is ExFAT...and a quick search indicates that v1.2 and up support ExFAT. It
    even supports NTFS.

    --
    _/_
    / v \ Scott Alfter (remove the obvious to send mail)
    (IIGS( https://alfter.us/ Top-posting!
    \_^_/ >What's the most annoying thing on Usenet?

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From druck@3:770/3 to R.Wieser on Friday, June 24, 2022 14:38:23
    On 18/06/2022 09:01, R.Wieser wrote:
    Scott,

    use gparted to shrink the image size to the minimum needed
    to create the image, and to expand into the full space of a
    new SD card once the image is dd'd onto it.

    gparted knows enough of the different filesystems to be able to do that ? Hmmmm... Maybe I should take a look at it.

    gparted knows about a large number of different filing systems and the
    tools required to do operations such as check and resize them.

    Although I like the idea of shrinking the backups to whats actually used, it also means I need to keep an un-shrunken (plain DD) backup with gparted on
    it to be able to do that ...

    I don't bother shrinking the backup partition files, the current nightly
    backup is stored uncompressed and I make compressed weekly and monthly
    copies.

    The trick being is to use the zerofree tool on the image file, to zero
    all the free space, then when you gzip that compresses away to nothing.
    It will be the same size as if you shrank the image then compressed it,
    but with the advantage unzipping will expand it back to the original
    image size.

    zerofree is also a lot faster than shuffling data when you shrink an
    image. Use pigz to compress and it will use all cores to zip it, far
    quicker than gzip. Finally you can gunzip -c the compressed image and
    pipe it in to a dd command writing straight to an SD card.

    ---druck

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)