← Back to team overview

group.of.nepali.translators team mailing list archive

[Bug 2060769] Re: pro-client decodes utf-8 strictly, which causes problems in some locales

 

This bug was fixed in the package ubuntu-advantage-tools - 33.2~22.04

---------------
ubuntu-advantage-tools (33.2~22.04) jammy; urgency=medium

  * Backport 33.2 to jammy (LP: #2069237)

ubuntu-advantage-tools (33.2) oracular; urgency=medium

  * d/apparmor: add apt-news access to package information on the system
    (LP: #2072489) (GH: #3193)

ubuntu-advantage-tools (33.1) oracular; urgency=medium

  * New upstream release 33.1: (LP: #2060769)
    - system:
      + always pass C.UTF8 as the language when calling a subprocess
      + ignore utf-8 decode errors on subprocess output

ubuntu-advantage-tools (33) oracular; urgency=medium

  * d/apparmor: adjust the esm_cache apparmor profile to allow reading of dpkg
    data directory (LP: #2067810) (GH: #3137)
  * New upstream release 33 (LP: #2069237)
    - apt: use Python bindings instead of apt CLI to query for installed
      packages (LP: #2060769) (LP: #2068744)
    - beta: drop support for beta services
    - contracts: add support for contracts which target a specific series
    - fips: change enable functionality to ensure all packages with a FIPS
      candidate are upgraded to the FIPS version (GH: #2667)
    - fix:
      + add the current_status field to the plan api return object
      + change recommended attach method to magic attach (GH: #3040)
    - livepatch: prefer the term 'coverage' instead of 'support' in messaging
      (GH: #3063)
    - realtime:
      + auto-select the raspi variant when appropriate
      + inform the user when auto-selecting a variant

 -- Lucas Moura <lucas.moura@xxxxxxxxxxxxx>  Thu, 18 Jul 2024 11:20:14
-0400

** Changed in: ubuntu-advantage-tools (Ubuntu Jammy)
       Status: Fix Committed => Fix Released

** Changed in: ubuntu-advantage-tools (Ubuntu Focal)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/2060769

Title:
  pro-client decodes utf-8 strictly, which causes problems in some
  locales

Status in ubuntu-advantage-tools package in Ubuntu:
  Fix Released
Status in ubuntu-advantage-tools source package in Xenial:
  Fix Released
Status in ubuntu-advantage-tools source package in Bionic:
  Fix Released
Status in ubuntu-advantage-tools source package in Focal:
  Fix Released
Status in ubuntu-advantage-tools source package in Jammy:
  Fix Released
Status in ubuntu-advantage-tools source package in Noble:
  Fix Released

Bug description:
  [ Impact ]

  If for any reason the user's system has any non-utf8 character as part
  of any subprocess command executed, the Pro Client will fail to parse
  the return output and break.

  Independent of why this happens, the Pro Client should not be relying
  on CLI output when there are Python bindings available to perform the
  same task - and python3-apt provides ways to list installed packages.

  The fix to this bug is to ignore utf-8 errors in the occasions where
  it needs to subprocess. Even other software than APT may include non-
  utf8 output by chance. To avoid corner case scenarios, setting LANG=C
  when executing any subprocess should mitigate the presence of non-utf8
  anyway.

  [ Test Plan ]

  Subprocessing is part of the regular operation of the Pro Client in
  many scenarios covered by the integration / acceptance tests.

  For this particular issue:
  - Launch a container for the release to be tested
  - Install a non-utf-8 locale and set it as the default. e.g.
      - sudo locale-gen fr_FR
      - sudo dpkg-reconfigure locales # select fr_FR
      - # log out and log in
  - pro status
  - see it fail
  - install the package from -proposed
  - see it does not fail anymore

  [ Where problems could occur ]

  Ignoring non-utf8 characters when decoding will not prevent the Client
  from reading all the actual utf-8 present in the output. The pieces we
  parse from the commands we execute are not involving any possible non-
  utf8 text. If we would have assessed this wrong, this could cause all
  sorts of regressions.

  To be extra careful, we are setting LANG=C.UTF-8 - so we have way
  better expectation management on the outputs from any external command
  we run on a system.

  [ Original description ]

  ["2024-04-10T08:28:17.952", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/os-release", {}]
  ["2024-04-10T08:28:17.953", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /proc/version_signature", {}]
  ["2024-04-10T08:28:18.575", "ERROR", "ubuntupro.cli", "wrapper", 1768, "Unhandled exception, please file a bug", {"exc_info": "Traceback (most recent call
   last):\n  File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1705, in wrapper\n    return func(*args, **kwargs)\n  File \"/usr/lib/py
  thon3/dist-packages/uaclient/cli/__init__.py\", line 1838, in main\n    return_value = args.action(args, cfg=cfg, extra_args=extra_args)\n  File \"/usr/li
  b/python3/dist-packages/uaclient/cli/__init__.py\", line 1475, in action_status\n    status, ret = actions.status(\n  File \"/usr/lib/python3/dist-package
  s/uaclient/actions.py\", line 173, in status\n    status = ua_status.status(cfg=cfg, show_all=show_all)\n  File \"/usr/lib/python3/dist-packages/uaclient/
  status.py\", line 418, in status\n    response = _unattached_status(cfg)\n  File \"/usr/lib/python3/dist-packages/uaclient/status.py\", line 283, in _unat
  tached_status\n    resources = get_available_resources(cfg)\n  File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 680, in get_available_re
  sources\n    resources = client.available_resources()\n  File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 114, in available_resources\n
     activity_info = self._get_activity_info()\n  File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 413, in _get_activity_info\n    \"deskt
  op\": system.is_desktop(),\n  File \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 397, in is_desktop\n    for package in apt.get_installed_pa
  ckages():\n  File \"/usr/lib/python3/dist-packages/uaclient/apt.py\", line 727, in get_installed_packages\n    out, _ = system.subp([\"apt\", \"list\", \"
  --installed\"])\n  File \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 703, in subp\n    out, err = _subp(\n  File \"/usr/lib/python3/dist-pa
  ckages/uaclient/system.py\", line 649, in _subp\n    out_result = out.decode(\"utf-8\") if out else \"\"\nUnicodeDecodeError: 'utf-8' codec can't decode b
  yte 0xe1 in position 109: invalid continuation byte"}]
  ["2024-04-10T08:30:25.404", "DEBUG", "ubuntupro.config", "parse_config", 625, "Using client configuration file at /etc/ubuntu-advantage/uaclient.conf", {}
  ]
  ["2024-04-10T08:30:25.408", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/ubuntu-advantage/uaclient.conf", {}]
  ["2024-04-10T08:30:25.409", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/user-config.json but file does not exis
  t", {}]
  ["2024-04-10T08:30:25.421", "DEBUG", "ubuntupro.http", "configure_web_proxy", 146, "Setting no_proxy: 169.254.169.254,[fd00:ec2::254],metadata", {}]
  ["2024-04-10T08:30:25.422", "DEBUG", "ubuntupro.http", "configure_web_proxy", 154, "Setting global proxy dict", {}]
  ["2024-04-10T08:30:25.422", "DEBUG", "ubuntupro.cli", "main", 1825, "Executed with sys.argv: ['/usr/bin/pro', 'attach', '<REDACTED>']", {}]
  ["2024-04-10T08:30:25.422", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/private/machine-token.json but file doe
  s not exist", {}]
  ["2024-04-10T08:30:25.422", "DEBUG", "ubuntupro.lock", "__enter__", 77, "spin lock starting for pro attach", {}]
  ["2024-04-10T08:30:25.423", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /run/ubuntu-advantage/notices/60-operation_in_progress atomicall
  y via tempfile /run/ubuntu-advantage/notices/tmparyyoirn", {}]
  ["2024-04-10T08:30:25.423", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /var/lib/ubuntu-advantage/lock atomically via tempfile /var/lib/
  ubuntu-advantage/tmpn1yxl91o", {}]
  ["2024-04-10T08:30:25.424", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /run/ubuntu-advantage/notices/60-operation_in_progress atomicall
  y via tempfile /run/ubuntu-advantage/notices/tmpwqyq85v1", {}]
  ["2024-04-10T08:30:25.424", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/private/machine-token.json but file doe
  s not exist", {}]
  ["2024-04-10T08:30:25.424", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/machine-id", {}]
  ["2024-04-10T08:30:25.424", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/os-release", {}]
  ["2024-04-10T08:30:25.425", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /proc/version_signature", {}]
  ["2024-04-10T08:30:25.970", "DEBUG", "ubuntupro.system", "ensure_file_absent", 576, "Removed file: /run/ubuntu-advantage/notices/60-operation_in_progress"
  , {}]
  ["2024-04-10T08:30:25.970", "DEBUG", "ubuntupro.system", "ensure_file_absent", 576, "Removed file: /var/lib/ubuntu-advantage/lock", {}]
  ["2024-04-10T08:30:25.971", "ERROR", "ubuntupro.cli", "wrapper", 1768, "Unhandled exception, please file a bug", {"exc_info": "Traceback (most recent call
   last):\n  File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1705, in wrapper\n    return func(*args, **kwargs)\n  File \"/usr/lib/py
  thon3/dist-packages/uaclient/cli/__init__.py\", line 1838, in main\n    return_value = args.action(args, cfg=cfg, extra_args=extra_args)\n  File \"/usr/li
  b/python3/dist-packages/uaclient/cli/__init__.py\", line 257, in new_f\n    return f(args, cfg=cfg, **kwargs)\n  File \"/usr/lib/python3/dist-packages/uac
  lient/cli/__init__.py\", line 202, in new_f\n    return f(*args, **kwargs)\n  File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 186,
  in new_f\n    retval = f(*args, cfg=cfg, **kwargs)\n  File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1305, in action_attach\n    a
  ctions.attach_with_token(cfg, token=token, allow_enable=allow_enable)\n  File \"/usr/lib/python3/dist-packages/uaclient/actions.py\", line 65, in attach_w
  ith_token\n    new_machine_token = contract_client.add_contract_machine(\n  File \"/usr/lib/python3/dist-packages/uaclient/util.py\", line 92, in decorato
  r\n    return f(*args, **kwargs)\n  File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 92, in add_contract_machine\n    activity_info = se
  lf._get_activity_info()\n  File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 413, in _get_activity_info\n    \"desktop\": system.is_deskt
  op(),\n  File \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 397, in is_desktop\n    for package in apt.get_installed_packages():\n  File \"/
  usr/lib/python3/dist-packages/uaclient/apt.py\", line 727, in get_installed_packages\n    out, _ = system.subp([\"apt\", \"list\", \"--installed\"])\n  Fi
  le \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 703, in subp\n    out, err = _subp(\n  File \"/usr/lib/python3/dist-packages/uaclient/syste
  m.py\", line 649, in _subp\n    out_result = out.decode(\"utf-8\") if out else \"\"\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position
  109: invalid continuation byte"}]
  ["2024-04-10T08:38:29.763", "DEBUG", "ubuntupro.config", "parse_config", 625, "Using client configuration file at /etc/ubuntu-advantage/uaclient.conf", {}
  ]
  ["2024-04-10T08:38:29.767", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/ubuntu-advantage/uaclient.conf", {}]
  ["2024-04-10T08:38:29.768", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/user-config.json but file does not exis
  t", {}]
  ["2024-04-10T08:38:29.780", "DEBUG", "ubuntupro.http", "configure_web_proxy", 146, "Setting no_proxy: 169.254.169.254,[fd00:ec2::254],metadata", {}]
  ["2024-04-10T08:38:29.780", "DEBUG", "ubuntupro.http", "configure_web_proxy", 154, "Setting global proxy dict", {}]
  ["2024-04-10T08:38:29.780", "DEBUG", "ubuntupro.cli", "main", 1825, "Executed with sys.argv: ['/usr/bin/pro', 'attach', '<REDACTED>']", {}]
  ["2024-04-10T08:38:29.781", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/private/machine-token.json but file doe
  s not exist", {}]
  ["2024-04-10T08:38:29.781", "DEBUG", "ubuntupro.lock", "__enter__", 77, "spin lock starting for pro attach", {}]
  ["2024-04-10T08:38:29.781", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /run/ubuntu-advantage/notices/60-operation_in_progress atomicall
  y via tempfile /run/ubuntu-advantage/notices/tmplx6ehdag", {}]
  ["2024-04-10T08:38:29.782", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /var/lib/ubuntu-advantage/lock atomically via tempfile /var/lib/
  ubuntu-advantage/tmp6ml_ltfj", {}]
  ["2024-04-10T08:38:29.782", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /run/ubuntu-advantage/notices/60-operation_in_progress atomicall
  y via tempfile /run/ubuntu-advantage/notices/tmpvxf_b1v0", {}]
  ["2024-04-10T08:38:29.783", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/private/machine-token.json but file doe
  s not exist", {}]
  ["2024-04-10T08:38:29.783", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/machine-id", {}]
  ["2024-04-10T08:38:29.783", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/os-release", {}]
  ["2024-04-10T08:38:29.784", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /proc/version_signature", {}]
  ["2024-04-10T08:38:30.366", "DEBUG", "ubuntupro.system", "ensure_file_absent", 576, "Removed file: /run/ubuntu-advantage/notices/60-operation_in_progress"
  , {}]
  ["2024-04-10T08:38:30.367", "DEBUG", "ubuntupro.system", "ensure_file_absent", 576, "Removed file: /var/lib/ubuntu-advantage/lock", {}]
  ["2024-04-10T08:38:30.367", "ERROR", "ubuntupro.cli", "wrapper", 1768, "Unhandled exception, please file a bug", {"exc_info": "Traceback (most recent call
   last):\n  File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1705, in wrapper\n    return func(*args, **kwargs)\n  File \"/usr/lib/py
  thon3/dist-packages/uaclient/cli/__init__.py\", line 1838, in main\n    return_value = args.action(args, cfg=cfg, extra_args=extra_args)\n  File \"/usr/li
  b/python3/dist-packages/uaclient/cli/__init__.py\", line 257, in new_f\n    return f(args, cfg=cfg, **kwargs)\n  File \"/usr/lib/python3/dist-packages/uac
  lient/cli/__init__.py\", line 202, in new_f\n    return f(*args, **kwargs)\n  File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 186,
  in new_f\n    retval = f(*args, cfg=cfg, **kwargs)\n  File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1305, in action_attach\n    a
  ctions.attach_with_token(cfg, token=token, allow_enable=allow_enable)\n  File \"/usr/lib/python3/dist-packages/uaclient/actions.py\", line 65, in attach_w
  ith_token\n    new_machine_token = contract_client.add_contract_machine(\n  File \"/usr/lib/python3/dist-packages/uaclient/util.py\", line 92, in decorato
  r\n    return f(*args, **kwargs)\n  File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 92, in add_contract_machine\n    activity_info = se
  lf._get_activity_info()\n  File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 413, in _get_activity_info\n    \"desktop\": system.is_deskt
  op(),\n  File \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 397, in is_desktop\n    for package in apt.get_installed_packages():\n  File \"/
  usr/lib/python3/dist-packages/uaclient/apt.py\", line 727, in get_installed_packages\n    out, _ = system.subp([\"apt\", \"list\", \"--installed\"])\n  Fi
  le \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 703, in subp\n    out, err = _subp(\n  File \"/usr/lib/python3/dist-packages/uaclient/syste
  m.py\", line 649, in _subp\n    out_result = out.decode(\"utf-8\") if out else \"\"\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position
  109: invalid continuation byte"}]

  ProblemType: Bug
  DistroRelease: Ubuntu 20.04
  Package: ubuntu-advantage-tools 31.2~20.04
  ProcVersionSignature: Ubuntu 5.15.0-101.111~20.04.1-generic 5.15.143
  Uname: Linux 5.15.0-101-generic x86_64
  NonfreeKernelModules: falcon_lsm_serviceable falcon_nf_netcontain falcon_kal falcon_lsm_pinned_16407 falcon_lsm_pinned_16206
  ApportVersion: 2.20.11-0ubuntu27.27
  Architecture: amd64
  CasperMD5CheckResult: skip
  Date: Wed Apr 10 08:28:11 2024
  PackageArchitecture: all
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   LANG=es_ES
   SHELL=/bin/bash
  SourcePackage: ubuntu-advantage-tools
  UpgradeStatus: No upgrade log present (probably fresh install)
  apparmor_logs.txt:

  cloud-id.txt-error: Invalid command specified 'cloud-id'.
  livepatch-status.txt-error: Invalid command specified '/snap/bin/canonical-livepatch status'.
  ua-status.json-error: Failed running command 'pro status --format json' [exit(1)]. Message:
  uaclient.conf:
   contract_url: https://contracts.canonical.com
   log_level: debug

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2060769/+subscriptions