Skip to content

WeeklyTelcon_20181016

Jump to bottom
Geoffrey Paulsen edited this page Jan 15, 2019 · 1 revision

Open MPI Weekly Telecon

  • Dialup Info: (Do not post to public mailing list or public wiki)

Attendees (in Person)

  • Geoff Paulsen
  • Jeff Squyres
  • Brian
  • Edgar Gabriel
  • Howard Pritchard --- (on Phone)
  • Ralph Castain
  • Todd Kordenbrock
  • David Bernholdt
  • Akshay Venkatesh
  • Nathan Hjelm

not there today (I keep this for easy cut-n-paste for future notes)

  • Arm (UTK)
  • Dan Topa (LANL)
  • Thomas Naughton
  • Aravind Gopalakrishnan (Intel)
  • Akvenkatesh (nVidia)
  • Geoffroy Vallee
  • Dan Topa (LANL)
  • George
  • Peter Gottesman (Cisco)
  • Josh Hursey
  • Joshua Ladd?
  • Matias Cabral
  • Matthew Dosanjh
  • Xin Zhao
  • mohan

Agenda/New Business

  • Vader - Compilers are wripping our word-sized writes.

    • compiler no longer guarantee that a word-sized write is an atomic write. They used to gauarantee, but no longer guarantee this.
    • Linux kernel solved this by having a contract with gcc and llvm on exactly what volatile meant so that access_once, Read_once, and Write_once macros works.
    • Other way to solve this is custom asm to prevent writes from being wripped.
    • Our Options:
      • Require the core part of Open MPI to be compiled with a compiler that honors what gcc does with volatile (gcc v4.0 and later, last few years of icc, and llvm)
    • On master, we use C11 atomics by default. If C11 isn't available, we'll use gcc sync builtins (no atomic load/store) finally we'd use our hand done atomics. HOPEFULLY we can just use C11, instead of gcc sync, and hand done ones, but
      • C11 - has atomic load/store
      • gcc builtins - has atomic load/store -sufficent
      • sync builtiuns - Dont work for us.
      • base assembly - very easy to write an atomic load/store.
    • The day we REQUIRE C11 is a glorious day. :)
    • Brian thinks we should reduce sync ops to below hand-asm priority
      • Then only enable fbox support if there is atomic load/store support
    • Do this mid-stream - if using sync built-ins today.
    • If on a platform without hand-asm, would still use sync built-ins, but disable vader-fastbox feature.
    • AS we do more and more with atomics rather than locks, it's going to be harder and harder to support such a wide range of compilers.
    • Would support a "wrapper compiler" that compiles MOST things, but compiles atomic based things with core compilers (smaller list that conforms)

  • Face to Face is next week

    • Oct 16th - Brian and Nathan might come this one day.
      • Libfabric / OFI on Oct16th.

  • github suggestion on email filtering

Minutes

Review All Open Blockers

Review v2.1.6 (not going to do this in immediate future.

  • PRing Nathan's Vader BTL for fastbox to ALL release branches back to v2.1.x

  • Compilers COULD but probably won't get around this fix, so it should be good.

  • Nathan will discuss some future vader fixes later.

  • This PR is good for Release branches.

  • Vader problem is still happening on i386 and MIPSL nodes.

    • Do we want to just NOT support 32bit builds?
    • That makes our packager's lives difficult.
    • 32bit should be considered a "canary in the coalmine", and we might have other REAL issues.
    • Tested with patch, and still failing, so THIS might not be the only issue.
    • Not ready to say "drop 32bit".
    • Brian will investigate as time permits.

  • Driving a new release because it's a regression.

  • Dec 1st.

Review v3.0.x Milestones v3.0.3

  • Schedule:
  • Friday (Do another RC for both 3.0 and 3.1)

Review v3.1.x Milestones v3.1.0

  • Going to Release RC in parallel with 3.0.x Friday

v4.0.0

  • Schedule: Ship (last?) RC tomorrow Wed 17th
  • Will do RC this week. Hopefully last?
  • No known
  • PR for sigchild.

PMIx

  • Nothing new at this time.

  • No action: Open MPI v5.x Future of Launch

    • Geoffroy Vallee sent out document with summary to core-devel.
      Everyone please read and reply.
    • ORTE/PRTE
      • We had a working group meeting to discuss launching under Open MPI v5.0
      • Summary is to throw away ORTE, and make calls directly to PMIx, and then use PRTE with an mpirun wrapper around PRTE.
    • Split this into two steps:
      1. Make PMIx a first class citizen - and call PMIx API directly.
        • When we added the opal PMIx layer, we added infrastructure, and we're talking about flipping that around, so internally Open MPI calls PMIx calls, and then other components might translate the PMIx calls to PMI1 or PMI2 or whatever else.
        • PMIx community operating as a "standard" for over a year or so now.
        • PMIx standard document is in progress.
        • Just doing this much, should make ORTE much more in-line with PRTE, and make bugfixing between the two much less.
      2. Packaging / Launcher.
        • PRTE is that far ahead of ORTE because it's painful to move them back.
        • Many don't want to have to download something different to launch.
      3. Will need to ponder and come to consensus at face to face.

Master

  • 10,000s of TCP failures at Cisco on master.

  • Brian found two sets of TCP problems that are multi-nic.

    1. add_procs adds the procs to the structure, but doesn't fully setup datastructure
    • Race condition at setup.
    • Short term workaround:
      • MPI_THREAD_MULTIPLE or multiple nics. Set the bit to setup the world at startup. This works around the race condition.
      • Not high enough priority for v4.0.0, put workaround into v4.0.1
      • Dropped connection.
    • For 5.0 Brian is interested in doing the full re-design work for this.
    • Why not just NOT have tcp btls disabled by default, and have user enable instead.
    1. A patch UTK put in change on master that caused silent error into a fatal error.
      • If you have two MPI processes and one rank called a fin, we silently handled that (because if we had to talk to it again, then
      • George is working on fault tollerance at PML layer.
      • Brian will work with George on a fix for master.

  • Thought about adding CI to check for new warnings.

    • warning count delta is gross.
    • It's a bunch of work, but easier when we're down to a few warnings.
    • a couple in pmix.

  • Super Computing Open MPI BOF, and PMIx BOF

  • MTT - * IBM syntax failures. (Build / Install)

    • PML UCX data
    • v4.0.x
    • String Stuff
      • Ralph and Brian did asprintf()
      • Jeff update strncat and strcpy and strncpy
      • What is left to do? Hacktober fest - strcat()
      • Don't want to restrict users from CALLING these in the future with undef
      • Some users were calling incorrectly.
      • Compilers complaining about not checking return code, so our wrapper helps.
      • Will start a Review

New topics

  • MTT License discussion - MTT needs to be de-GPL-ified.

    • Main desire is python is in a repo with no GPL code (no Perl code)
    • Current status:
      • Need to make progress on sooner than later.
      • Ralph will move existing MTT to new mtt-legacy repo,
        • then rip out perl from MTT repo.
      • Cisco spins up a different slurm job for each MPI build, with a single ini file. By doing it this way, it depends on many perl funclets.
      • If change to have a different ini for each different "stream", it should work okay with python. Didn't happen before Peter left.
    • Ralph is waiting for MTT users to move to MTT-legacy repo.
      • Absoft, Amazon, IBM, need to move.

  • MTT performance database?

    • No status for a while.
    • MTT does report this, but no one looks.
    • Howard suggests many different performance dashboards.
      • Influx DB with jenkins, and can be queried.
      • Still need to get an up to date viewer.

Review Master Master Pull Requests

  • didn't discuss today.

Review Master MTT testing

Oldest PR

Oldest Issue

Status Updates:

Status Update Rotation

  1. Mellanox, Sandia, Intel
  2. LANL, Houston, IBM, Fujitsu
  3. Amazon,
  4. Cisco, ORNL, UTK, NVIDIA

Back to 2018 WeeklyTelcon-2018

Clone this wiki locally