[Bioc-devel] Bioconductor Git/GitHub Mirrors

Fri Jun 19 22:50:07 CEST 2015

Hi,

Dan previously said:

Try starting over again. Remove your local repository and do a fresh clone:

git clone https://github.com/leekgroup/derfinderHelper.git
cd derfinderHelper
bash /path/to/update_remotes.sh
git checkout devel
git svn rebase
git merge master --no-edit
git svn dcommit --add-author-from

So I went and did that. Note that if you have local branches, like I
did, make sure that you push them to GitHub first.

1) I first encountered an authentification error. I solved it below by
deleting any previous SVN auth info I had (I only use SVN for Bioc).

$ git svn dcommit --add-author-from --username=l.collado-torres
Committing to https://hedgehog.fhcrc.org/bioconductor/trunk/madman/Rpacks/derfinderHelper
...

ERROR from SVN:
URL access forbidden for unknown reason: Access to
'/bioconductor/!svn/me' forbidden
No changes between ef1c7dc3236ba31adace8c2205d30cc76e913032 and
refs/remotes/git-svn-devel
Resetting to the latest refs/remotes/git-svn-devel
ERROR: Not all changes have been committed into SVN, however the committed
ones (if any) seem to be successfully integrated into the working tree.
Please see the above messages for details.

Solution:

## Check what auth files I had
$ ls  ~/.subversion/auth/svn.simple/*

## Was my username in any of them? Nope
$ grep -l l.collado-torres  ~/.subversion/auth/svn.simple/*

## Clear them. If you use SVN for multiple repos, you don't want to do this.
$ rm  ~/.subversion/auth/svn.simple/*

## Yup, it's empty.
$ ls  ~/.subversion/auth/svn.simple/*

## Start over and use this line on the first git svn rebase
$ git svn rebase --username=l.collado-torres

## Now my SVN auth info is there
$ ls  ~/.subversion/auth/svn.simple/*
$ grep -l l.collado-torres  ~/.subversion/auth/svn.simple/*

2) The commands Dan posted before worked just like he described. That
is, it didn't commit anything back to SVN (since there's nothing new).
I'm re-posting them for clarity.

## Remove local repository and do a fresh clone
git clone https://github.com/leekgroup/derfinderHelper.git
cd derfinderHelper
bash /path/to/update_remotes.sh
git checkout devel
git svn rebase
git merge master --no-edit
git svn dcommit --add-author-from

## Output from last command
Committing to https://hedgehog.fhcrc.org/bioconductor/trunk/madman/Rpacks/derfinderHelper
...
No changes
4a21725268e160061103dadb063d0f382235d7e6~1 ==
4a21725268e160061103dadb063d0f382235d7e6
No changes between 4a21725268e160061103dadb063d0f382235d7e6 and
refs/remotes/git-svn-devel
Resetting to the latest refs/remotes/git-svn-devel

As Dan pointed out earlier, I do not want to make my "master" commit
history the one from SVN. Want I want to achieve is to have my past
commit history be the one from GitHub, and then after that, I don't
mind the double commit messages like in
https://github.com/rmflight/categoryCompare/commit/762bf7046931096b730e21d8a9c4d0d02c734602
and https://github.com/rmflight/categoryCompare/commit/e132385d0ada77bb9a43e1f7e0027f6b6d59af19

3) With my fresh clone, if I follow the original instructions, I still
end with the problem I described in my first email.

$ git clone https://github.com/leekgroup/derfinderHelper.git
Cloning into 'derfinderHelper'...
remote: Counting objects: 387, done.
remote: Total 387 (delta 0), reused 0 (delta 0), pack-reused 387
Receiving objects: 100% (387/387), 81.50 KiB | 0 bytes/s, done.
Resolving deltas: 100% (214/214), done.
Checking connectivity... done.
cd derfinderHelper

$ bash ../update_remotes.sh
Commit to git as normal, when you want to push your commits to svn
  1. `git checkout devel` to switch to the devel branch. (use release-X.X for
        release branches)
  2. `git svn rebase` to get the latest SVN changes.
  3. `git merge master --no-edit` to merge your changes from the master branch
        or skip this step and work directly on the current branch.
  4. `git svn rebase && git svn dcommit --add-author-from` to sync and commit
        your changes to svn.

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

$ git checkout devel
Switched to branch 'devel'
Your branch is up-to-date with 'bioc/master'.

$ git svn rebase
Migrating from a git-svn v1 layout...
Data from a previous version of git-svn exists, but
.git/svn
(required for this version (2.4.3) of git-svn) does not exist.
Done migrating from a git-svn v1 layout
Rebuilding .git/svn/refs/remotes/git-svn-devel/.rev_map.bc3139a8-67e5-0310-9ffc-ced21a209358
...
r94975 = 89b8467a9e02d9f1d8ad2396c8d64cd44babd726
r95029 = 2372c1563d70030f278f2bf265d83e3013e67611
r95439 = d781eefa74f756c74561adf95bb638aa90935cc6
r95443 = 80527e2b9c1442e4d60127fd6d3f8a84cdb4a5db
r95451 = cf659b64cca6bac275f7c15f93c9ab1407800aa7
r95474 = 4f5d77e2ea1427d0dbb4a4ffad163f7ad58facc4
r95475 = 31f6cf4046778942c9357a6378928652559217ff
r96122 = e02c586ea34f902b4c068bcbe1d2d10d0830a00b
r96159 = ed98baccbec23e0f70d8aba6a1eb11941c844081
r96165 = f56cb64e2fbd0cb7e4580047c1f27eb8400db67e
r96168 = d8b8901ccfa13d30b8e9ecc5b195e7f6842a2194
r96450 = c5f4ad07ff97d733f36a1f2a6556fdbab7c46459
r96458 = 2f027fc7ee011b579bc51cc6bb062987fafb16cd
r96463 = 97f90df0ec9cf1a6131bfd4479c6cca0981fa72c
r96466 = a193e8c3ca4650a87b0f61416f0cd32169c193df
r96488 = e3bb06e026e8890e9161d6504e208273998d74c5
r97003 = f2be8f26f80b369f14df9cd4db37b573db71a8e3
r97073 = 2698148110d3b45ef335e3ce1ce555af7459bbaa
r100624 = 4cc67a757cd51ce529575197ab7dddcf7dfde286
r102591 = 9b856813fa041972a398ddc22362bf3c52789d5b
r102594 = 441927a8a2d41ea0c7f0db7f05e1c2600a45fe0d
r104736 = 601760f4c3fffbd897fa4f8fa3a631dbf8d0dda8
r104883 = 1a2fcc1b1a63b3e7398e5d9001f6c358369d6161
r104889 = cb528993af8e7d9d4d41a17027dc64aec84e0255
r104895 = ace63767281447e9497876c700f3538f28d238d9
r104898 = 08ec97478d7836265db955f3032c28490b212488
r104900 = b452efe2f8923c5f2229feecc1b399d56364d52d
r104916 = 3e54179b27b9641cff04f699fb5b915e45347b94
r104998 = 3e8de0408d30bea6b06163a63d179dec5456eaa8
Done rebuilding
.git/svn/refs/remotes/git-svn-devel/.rev_map.bc3139a8-67e5-0310-9ffc-ced21a209358
Current branch devel is up to date.

$ git merge master --no-edit
Merge made by the 'recursive' strategy.

$ git status
On branch devel
Your branch is ahead of 'bioc/master' by 73 commits.
  (use "git push" to publish your local commits)
nothing to commit, working directory clean

$ git svn rebase && git svn dcommit --add-author-from
First, rewinding head to replay your work on top of it...
Applying: Initial commit
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging README.md
CONFLICT (add/add): Merge conflict in README.md
Failed to merge in the changes.
Patch failed at 0001 Initial commit
The copy of the patch that failed is found in:
   /Users/lcollado/Dropbox/JHSPH/Code/derfinderHelper/.git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".

rebase refs/remotes/git-svn-devel: command returned error: 1

$ git status
rebase in progress; onto 3e8de04
You are currently rebasing branch 'devel' on '3e8de04'.
  (fix conflicts and then run "git rebase --continue")
  (use "git rebase --skip" to skip this patch)
  (use "git rebase --abort" to check out the original branch)

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   LICENSE

Unmerged paths:
  (use "git reset HEAD <file>..." to unstage)
  (use "git add <file>..." to mark resolution)

both added:      README.md

$ git rebase --abort

$ git status
On branch devel
Your branch is ahead of 'bioc/master' by 73 commits.
  (use "git push" to publish your local commits)
nothing to commit, working directory clean

$ git svn dcommit --add-author-from
Committing to https://hedgehog.fhcrc.org/bioconductor/trunk/madman/Rpacks/derfinderHelper
...
No changes
750d5e459ae6f882922144a6ba4641cdc8ba84b3~1 ==
750d5e459ae6f882922144a6ba4641cdc8ba84b3
No changes between 750d5e459ae6f882922144a6ba4641cdc8ba84b3 and
refs/remotes/git-svn-devel
Resetting to the latest refs/remotes/git-svn-devel
$ git status
On branch devel
Your branch is up-to-date with 'bioc/master'.
nothing to commit, working directory clean

4) In an attempt to try to achieve my goal, I go back to the state
after the merge (that is, after one git svn rebase). That is:

## Remove local repository and do a fresh clone
git clone https://github.com/leekgroup/derfinderHelper.git
cd derfinderHelper
bash /path/to/update_remotes.sh
git checkout devel
git svn rebase
git merge master --no-edit

I know the second git svn rebase creates the conflicts, but I thought
from http://stackoverflow.com/questions/2959443/why-is-the-meaning-of-ours-and-theirs-reversed-with-git-svn
that I could get git svn to recursively choose my git history over my
svn history. However, it doesn't work as shown below. From
http://git-scm.com/docs/git-svn, under the "rebase" command, it says
"This accepts all options that git svn fetch and git rebase accept."
Below I try a couple of different syntaxes which I thought would work
given http://git-scm.com/docs/git-rebase

$ git svn rebase -s strategy recursive -X theirs
Unknown option: X
$ git svn rebase -s strategy "recursive" -X "theirs"
Unknown option: X
$ git svn rebase -s strategy recursive -Xtheirs
Unknown option: X
Unknown option: t
Unknown option: e
$ git svn rebase -s strategy recursive --strategy-option theirs
Unknown option: strategy-option

$ git svn rebase -s strategy "recursive -X theirs"
First, rewinding head to replay your work on top of it...
/usr/local/Cellar/git/2.4.3/libexec/git-core/git-rebase--merge: line
70: git-merge-strategy: command not found
Unknown exit code (127) from command: git-merge-strategy
ed5ceda167117937ea498d8713361c34e2fb84e3^ -- HEAD
ed5ceda167117937ea498d8713361c34e2fb84e3
rebase --strategy=strategy refs/remotes/git-svn-devel: command returned error: 1
$ git rebase --abort

$ git svn rebase -s theirs
First, rewinding head to replay your work on top of it...
/usr/local/Cellar/git/2.4.3/libexec/git-core/git-rebase--merge: line
70: git-merge-theirs: command not found
Unknown exit code (127) from command: git-merge-theirs
ed5ceda167117937ea498d8713361c34e2fb84e3^ -- HEAD
ed5ceda167117937ea498d8713361c34e2fb84e3
rebase --strategy=theirs refs/remotes/git-svn-devel: command returned error: 1
$ git rebase --abort

I know https://github.com/leekgroup/derfinderHelper/commit/ed5ceda167117937ea498d8713361c34e2fb84e3
is the first commit in my git history and
https://github.com/Bioconductor-mirror/derfinderHelper/commit/3e8de0408d30bea6b06163a63d179dec5456eaa8is
the last commit in the svn history.

What frustrates me the most at this point is that the opposite
strategy ("ours") works:

## Remove local repository and do a fresh clone
git clone https://github.com/leekgroup/derfinderHelper.git
cd derfinderHelper
bash /path/to/update_remotes.sh
git checkout devel
git svn rebase
git merge master --no-edit

$ git svn rebase -s ours
First, rewinding head to replay your work on top of it...
Already applied: 0001 Initial commit
Already applied: 0002 Delete LICENSE
Already applied: 0003 Imported derfinder::fstats.apply(). Having the
function in a separate package should speed up the permutation cycles
when using BiocParallel::SnowParam()
Already applied: 0004 minor formatting changes
Already applied: 0005 Fix links

## more output

Already applied: 0066 Change to leekgroup in vignettes
Already applied: 0067 Make startup messages very rare
Already applied: 0068 Fix to make messages very rare
All done.

Inspecting with "git log", I only see commits from the svn history.
The opposite of what I wanted, and hence why I want to use "theirs".
Given the syntax that works with "ours", I would expect
$ git svn rebase -s theirs
to work. Hm...

That lead me to
http://stackoverflow.com/questions/173919/is-there-a-theirs-version-of-git-merge-s-ours
but it didn't apply to this problem because that's for merging, not
rebasing. During all of this, I noticed that I could manually resolve
the conflicts using "theirs" via the SourceTree app. Sadly, I don't
know how to get the command SourceTree runs behind the scene but it
lead me to believe that I could manually choose for all the
conflicting files to use "theirs", then add them, then continue
rebasing, etc. That's how I found
http://stackoverflow.com/questions/24743769/git-resolve-conflict-using-ours-theirs-for-all-files

and...

## Remove local repository and do a fresh clone
git clone https://github.com/leekgroup/derfinderHelper.git
cd derfinderHelper
bash /path/to/update_remotes.sh
git checkout devel
git svn rebase
git merge master --no-edit
git svn dcommit --add-author-from
git svn rebase

## For each conflict loop, do
grep -lr '<<<<<<<' . | xargs git checkout --theirs
git add * ## If at some point you added files starting with .
## you will have to add them manually
git rebase --continue

## Once that's done, do
git svn dcommit --add-author-from

## Then pull from the Bioc mirror
git pull

In my case, that did:

$ git svn dcommit --add-author-from
Committing to https://hedgehog.fhcrc.org/bioconductor/trunk/madman/Rpacks/derfinderHelper
...
A LICENSE
M README.md
Committed r105182
A LICENSE
M README.md
r105182 = 07e2529c91e3bd6f12fc798ac39e4120a4f8154b (refs/remotes/git-svn-devel)
D LICENSE

## more output

Committed r105249
M R/zzz.R
r105249 = d35a0af365c1b7417564b598611371acd3dd7469 (refs/remotes/git-svn-devel)
No changes between 9fa56b177a184b18f75c8636a758ce18770831df and
refs/remotes/git-svn-devel
Resetting to the latest refs/remotes/git-svn-devel

 $ git status
On branch devel
Your branch is ahead of 'bioc/master' by 68 commits.
  (use "git push" to publish your local commits)
nothing to commit, working directory clean
$ git pull
>From https://github.com/Bioconductor-mirror/derfinderHelper
   3e8de04..d35a0af  master     -> bioc/master
Already up-to-date.

And now https://github.com/Bioconductor-mirror/derfinderHelper looks
like I wanted it to look. It has a single SVN commit per git commit I
originally did. Plus the original commits made directly to SVN by the
git-svn bridge and others at Bioconductor.

Yay!!!

5) Everything should be in sync.... well, kind of.

So I went back to my master branch, did a new commit to test things. Then...

git checkout devel
git svn rebase
git merge master ## I had some conflicts to resolve here
git svn rebase

Let me back to that issue with the first commit. Oh no!!

However, Dan's earlier solution did work. That is, after making a new
commit on the master branch

git checkout devel
git svn rebase
git merge master
git svn dcommit --add-author-from

Or it did work partially since
https://github.com/Bioconductor-mirror/derfinderHelper/commit/402100dc8d78ccbb99a4d65dc6f748fb7474a69d
does not include the message from
https://github.com/leekgroup/derfinderHelper/commit/4b874a895701fe81c108cb4c98d07d0177ebcacd

 At this point, I think the best for me would be to make my "master"
branch have the same history as "devel" (aka master from
https://github.com/Bioconductor-mirror/derfinderHelper) (my git
history is there, just under the svn style of commits and with a
different username) and continue from there like
https://github.com/rmflight/categoryCompare is doing. Before doing so,
I'll save my current "master" into another branch and push it to
GitHub, just as a backup.

Otherwise, for every new commit, I'll have to re-write the full SVN
commit history and that will take too long.

Ugh... that was huge.

In summary, if this is ok, I plan for each of my other pkgs to:

## Make a copy of the "master" branch
## Push local branches to Github
## Remove local repository and do a fresh clone
git clone https://github.com/myUsername/MyPackage.git
cd MyPackage
bash /path/to/update_remotes.sh
git checkout devel
git svn rebase
git merge master --no-edit
git svn dcommit --add-author-from
git svn rebase

## For each conflict loop, do
grep -lr '<<<<<<<' . | xargs git checkout --theirs
git add *
git rebase --continue

## Once that's done, do
git svn dcommit --add-author-from

## Then pull from the Bioc mirror
git pull
## Delete the master branch
## Make a new master branch which is a copy from the devel branch

## Continue as https://github.com/rmflight/categoryCompare is doing

Cheers,
Leo

On Fri, Jun 19, 2015 at 2:27 PM, Dan Tenenbaum <dtenenba at fredhutch.org> wrote:
>
>
> ----- Original Message -----
>> From: "Guangchuang Yu" <gcyu at connect.hku.hk>
>> To: bioc-devel at r-project.org
>> Sent: Wednesday, June 17, 2015 5:04:04 AM
>> Subject: Re: [Bioc-devel] Bioconductor Git/GitHub Mirrors
>>
>> Dear Dan,
>>
>> I update my package
>> GOSemSim(https://github.com/GuangchuangYu/GOSemSim),
>> with the following steps:
>>
>> 1. delete the git-svn-bridge
>> 1. bash /path/to/update_remotes.sh
>> 2. git checkout devel
>> 3.   git checkout master
>> 4. git merge devel
>>
>> All was fine without any error.
>>
>
> I asked Jim about this and he said this last step may have made git svn fail to find the svn information. You can do this command here:
>
> git reset --hard a3ff1e7a3efd09b0c
>
> That's the commit before the merge. You should then be able to run "git svn dcommit".
>
>
> Dan
>
>
>> But when I follow the instruction to test a commit to svn.
>>
>>    1. Each time you want to push git commits to svn:
>>       1. git checkout devel to switch to the devel branch. (use
>>       release-X.X
>>       for release branches)
>>       2. git svn rebase to get the latest SVN changes.
>>       3. git merge master --no-edit to merge your changes from the
>>       master
>>       branch or skip this step and work directly on the current
>>       branch.
>>       4. git svn rebase && git svn dcommit --add-author-from to sync
>>       and
>>       commit your changes to svn. You may be prompted here for your
>> SVN username
>>       and password.
>>
>>
>> ➜  GOSemSim git:(master) git checkout devel
>> Switched to branch 'devel'
>> Your branch is up-to-date with 'bioc/master'.
>> ➜  GOSemSim git:(devel) git svn rebase
>> Migrating from a git-svn v1 layout...
>> Data from a previous version of git-svn exists, but
>>     .git/svn
>>     (required for this version (2.4.1) of git-svn) does not exist.
>> Done migrating from a git-svn v1 layout
>> Rebuilding
>> .git/svn/refs/remotes/git-svn-devel/.rev_map.bc3139a8-67e5-0310-9ffc-ced21a209358
>> ...
>> r35719 = 714adc6875914b83b673bcf93f6eac976676388d
>> r36178 = 199736e1f199d80a16121c0dca7e4aac015140bc
>> ...
>> ...
>> r104302 = b7770b576acd1439d7e01a5115866fe6bb84d2d7
>> r104400 = a3ff1e7a3efd09b0c0a760f2d4053331ee8039d2
>> Done rebuilding
>> .git/svn/refs/remotes/git-svn-devel/.rev_map.bc3139a8-67e5-0310-9ffc-ced21a209358
>> Current branch devel is up to date.
>>
>>   GOSemSim git:(devel) git merge master --no-edit
>> Updating a3ff1e7..5b9b679
>> Fast-forward
>> ➜  GOSemSim git:(devel) git svn rebase && git svn dcommit
>> --add-author-from
>> Unable to determine upstream SVN information from working tree
>> history
>>
>>
>> It throw the above error. Any idea?
>>
>> Bests,
>> Guangchuang
>> --
>> --~--~---------~--~----~------------~-------~--~----~
>> Guangchuang Yu, PhD Candidate
>> State Key Laboratory of Emerging Infectious Diseases
>> School of Public Health
>> The University of Hong Kong
>> Hong Kong SAR, China
>> www: http://ygc.name
>> -~----------~----~----~----~------~----~------~--~---
>>
>>       [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> Bioc-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel